From nad at python.org  Tue Nov  1 00:55:41 2016
From: nad at python.org (Ned Deily)
Date: Tue, 1 Nov 2016 00:55:41 -0400
Subject: [RELEASE] Python 3.6.0b3 is now available
Message-ID: <A534CD7D-C35F-4A2D-B491-A51431174746@python.org>

On behalf of the Python development community and the Python 3.6 release
team, I'm pleased to announce the availability of Python 3.6.0b3. 3.6.0b3
is the third of four planned beta releases of Python 3.6, the next major
release of Python.

Among the new major new features in Python 3.6 are:

* PEP 468 - Preserving the order of **kwargs in a function
* PEP 487 - Simpler customization of class creation
* PEP 495 - Local Time Disambiguation
* PEP 498 - Literal String Formatting
* PEP 506 - Adding A Secrets Module To The Standard Library
* PEP 509 - Add a private version to dict
* PEP 515 - Underscores in Numeric Literals
* PEP 519 - Adding a file system path protocol
* PEP 520 - Preserving Class Attribute Definition Order
* PEP 523 - Adding a frame evaluation API to CPython
* PEP 524 - Make os.urandom() blocking on Linux (during system startup)
* PEP 525 - Asynchronous Generators (provisional)
* PEP 526 - Syntax for Variable Annotations (provisional)
* PEP 528 - Change Windows console encoding to UTF-8 (provisional)
* PEP 529 - Change Windows filesystem encoding to UTF-8 (provisional)
* PEP 530 - Asynchronous Comprehensions

Please see "What?s New In Python 3.6" for more information:

https://docs.python.org/3.6/whatsnew/3.6.html

You can find Python 3.6.0b3 here:

https://www.python.org/downloads/release/python-360b3/

Beta releases are intended to give the wider community the opportunity
to test new features and bug fixes and to prepare their projects to
support the new feature release. We strongly encourage maintainers of
third-party Python projects to test with 3.6 during the beta phase and
report issues found to bugs.python.org as soon as possible. While the
release is feature complete entering the beta phase, it is possible that
features may be modified or, in rare cases, deleted up until the start
of the release candidate phase (2016-12-05). Our goal is have no changes
after rc1. To achieve that, it will be extremely important to get as
much exposure for 3.6 as possible during the beta phase. Please keep in
mind that this is a preview release and its use is not recommended for
production environments

The next pre-release of Python 3.6 will be 3.6.0b4, currently
scheduled for 2016-11-21. More information about the release schedule
can be found here:

https://www.python.org/dev/peps/pep-0494/

--
  Ned Deily
  nad at python.org -- []



From best_lay at yahoo.com  Tue Nov  1 01:00:52 2016
From: best_lay at yahoo.com (Wildman)
Date: Tue, 01 Nov 2016 00:00:52 -0500
Subject: Call a shell command from Python
References: <bOWdnTgDbsjFW4vFnZ2dnUU7-LPNnZ2d@giganews.com>
 <85r36x9lgi.fsf@benfinney.id.au>
 <mailman.19.1477889083.17234.python-list@python.org>
 <It2dnWjna7N5xIrFnZ2dnUU7-RednZ2d@giganews.com>
 <85mvhk9fbv.fsf@benfinney.id.au>
 <mailman.31.1477962586.17234.python-list@python.org>
Message-ID: <GdCdnZree_YZvIXFnZ2dnUU7-K-dnZ2d@giganews.com>

On Tue, 01 Nov 2016 12:08:52 +1100, Ben Finney wrote:

> Wildman via Python-list <python-list at python.org> writes:
> 
>> On Mon, 31 Oct 2016 15:44:13 +1100, Ben Finney wrote:
>>
>> > One immediate difference I see is that you specify different
>> > arguments to ?grep?. You have a different pattern for each command.
>> > 
>> > * The ?^user\:? pattern matches ?user\:? at the start of a line.
>> > 
>> > * The ?^$USER\:? pattern I think won't match anything, since ?$? matches
>> >   end-of-line and then you expect further characters *past* the end of
>> >   the line. I think that will always fail to match any line.
>>
>> Yes, the '^' indicates the start of the line and the ':' indicates
>> the character where to stop.  The colon has a special meaning so it
>> has to be escaped, '\:'.  The dollar sign precedes a variable.  In
>> this case it is an environment variable.
> 
> The ?grep? program you're invoking knows nothing of such variables, and
> the ?$? sign means to ?grep? what I said above.

You are correct about that but, in this case grep never "sees" the '$'
sign.  Bash expands $USER to the actual user name beforehand.  If you
are on a Linux system, enter this into a terminal to illustrate:

sudo grep ^$USER\: /etc/shadow

If the user name is ben then grep would see this:

grep ben\: /etc/shadow

>> > Maybe you are expecting Bash to be involved somehow (and so ?$USER?
>> > will be substituted by Bash with some other value). That's not what
>> > happens.
>>
>> No, the shell is already running.
> 
> I don't know what you mean by this. If you mean that some *other*
> instances of the shell ar running: that isn't relevant to how your
> Python program invokes a subprocess.

I simply meant that the script is run from a terminal.

> The shell is not involved in the command as you invoke it directly as a
> subprocess, without asking for a shell.
> 
>> And $USER will be substituted by the name of the user that invoked the
>> shell.
> 
> It will not, because there is no shell involved: your Python program
> invokes ?sudo?, which invokes ?grep?. The shell is never involved in
> that chain, so its substitutions rules are irrelevant.

I think my terminology is causing confusion.  I apologize for that. 

-- 
<Wildman> GNU/Linux user #557453


From ben+python at benfinney.id.au  Tue Nov  1 01:23:08 2016
From: ben+python at benfinney.id.au (Ben Finney)
Date: Tue, 01 Nov 2016 16:23:08 +1100
Subject: Call a shell command from Python
References: <bOWdnTgDbsjFW4vFnZ2dnUU7-LPNnZ2d@giganews.com>
 <85r36x9lgi.fsf@benfinney.id.au>
 <mailman.19.1477889083.17234.python-list@python.org>
 <It2dnWjna7N5xIrFnZ2dnUU7-RednZ2d@giganews.com>
 <85mvhk9fbv.fsf@benfinney.id.au>
 <mailman.31.1477962586.17234.python-list@python.org>
 <GdCdnZree_YZvIXFnZ2dnUU7-K-dnZ2d@giganews.com>
Message-ID: <85h97rai4j.fsf@benfinney.id.au>

Wildman via Python-list <python-list at python.org> writes:

> [?] in this case grep never "sees" the '$' sign. Bash expands $USER to
> the actual user name beforehand.

I understand how Bash substitutes variables on the command line.

What I need to repeat, though: In this case, no, Bash doesn't do that
because Bash isn't getting involved. Grep does in fact see the ?$? in
the command argument, because nothing ever replaces it.

> >> No, the shell is already running.
> > 
> > I don't know what you mean by this.
>
> I simply meant that the script is run from a terminal.

Which means that shell isn't involved in that command you're invoking
from inside the script.

The Bash instance which invoked your script is now *doing nothing*, and
will never do anything again until control returns to it (by your script
exiting, or suspending, or some other control interruption).

Your script invoking further programs will *not* get the earlier Bash
involved in that process (except in the special cases where those
programs themselves manipulate running processes; ?sudo? and ?grep? are
not special in this way).

So the way your script was invoked has no bearing on whether Bash will
get involved in what your script does. Your script is *directly*
invoking programs, and if you don't ask for a shell to be involved you
won't get it.

-- 
 \          ?? a Microsoft Certified System Engineer is to information |
  `\     technology as a McDonalds Certified Food Specialist is to the |
_o__)                               culinary arts.? ?Michael Bacarella |
Ben Finney



From steve+python at pearwood.info  Tue Nov  1 01:52:18 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Tue, 01 Nov 2016 16:52:18 +1100
Subject: Call a shell command from Python
References: <bOWdnTgDbsjFW4vFnZ2dnUU7-LPNnZ2d@giganews.com>
 <85r36x9lgi.fsf@benfinney.id.au>
 <mailman.19.1477889083.17234.python-list@python.org>
 <It2dnWjna7N5xIrFnZ2dnUU7-RednZ2d@giganews.com>
 <85mvhk9fbv.fsf@benfinney.id.au>
 <mailman.31.1477962586.17234.python-list@python.org>
 <GdCdnZree_YZvIXFnZ2dnUU7-K-dnZ2d@giganews.com>
Message-ID: <58182d94$0$22142$c3e8da3$5496439d@news.astraweb.com>

On Tue, 1 Nov 2016 04:00 pm, Wildman wrote:

> You are correct about that but, in this case grep never "sees" the '$'
> sign.  Bash expands $USER to the actual user name beforehand.  If you
> are on a Linux system, enter this into a terminal to illustrate:
> 
> sudo grep ^$USER\: /etc/shadow

Bash is not involved here. Python is calling grep directly.


You don't have to believe us, you can test this yourself. Create a simple
text file with a single line containing your username, and a simple Python
script that calls grep as you have been:


[steve at ando ~]$ echo $USER
steve
[steve at ando ~]$ cat foo.txt
blah blah steve blah blah
[steve at ando ~]$ cat greptest.py
import subprocess
cmdlist = ['grep', '$USER', 'foo.txt']
p = subprocess.Popen(cmdlist, stdout=subprocess.PIPE,stderr=subprocess.PIPE)
line, err = p.communicate()
print err, line

[steve at ando ~]$ python2.7 greptest.py

[steve at ando ~]$



So there you have it: categorical proof that bash does not expand the
string '$USER'. It cannot: bash is not involved in the subprocess call.
Python calls grep directly.

If you want to expand the '$USER' string from Python, do it yourself:


py> import os
py> os.environ['USER']
'steve'


> If the user name is ben then grep would see this:
> 
> grep ben\: /etc/shadow

If would, if you called grep from bash. But you didn't.


>>> > Maybe you are expecting Bash to be involved somehow (and so ?$USER?
>>> > will be substituted by Bash with some other value). That's not what
>>> > happens.
>>>
>>> No, the shell is already running.
>> 
>> I don't know what you mean by this. If you mean that some *other*
>> instances of the shell ar running: that isn't relevant to how your
>> Python program invokes a subprocess.
> 
> I simply meant that the script is run from a terminal.

That's irrelevant. Just because the script is running from a terminal
doesn't mean that the shell can peer deep inside each and every process and
magically apply the shell's string expansion rules.





-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From jobmattcon at gmail.com  Tue Nov  1 02:32:34 2016
From: jobmattcon at gmail.com (meInvent bbird)
Date: Mon, 31 Oct 2016 23:32:34 -0700 (PDT)
Subject: how to compile this code
Message-ID: <eae2aa23-0be0-4322-9245-609ec85c3859@googlegroups.com>

would like to change Add operator to custom function op2 in solve function
and then this solve([x*y - 1, x + 2], x, y)
during solve, the parameters also change Add to custom function op2

import ast 
from __future__ import division
from sympy import *
x, y, z, t = symbols('x y z t')
k, m, n = symbols('k m n', integer=True)
f, g, h = symbols('f g h', cls=Function)
import inspect

def op2(a,b): 
    return a*b+a 

class ChangeAddToMultiply(ast.NodeTransformer): 
    """Wraps all integers in a call to Integer()""" 
    def visit_BinOp(self, node): 
        print(dir(node))
        print(dir(node.left))
        if isinstance(node.op, ast.Add): 
            node.op = op2(node.left, node.right)
        return node 

code = inspect.getsourcelines(solve)
tree = ast.parse(code)
tree = ChangeAddToMultiply().visit(tree) 
ast.fix_missing_locations(tree) 
co = compile(tree, '<ast>', "exec") 

exec(code) 
exec(co) 


From jobmattcon at gmail.com  Tue Nov  1 03:40:06 2016
From: jobmattcon at gmail.com (meInvent bbird)
Date: Tue, 1 Nov 2016 00:40:06 -0700 (PDT)
Subject: how to debug this distributed program?
Message-ID: <9874b2d1-25b1-4d73-a96f-906f9584f750@googlegroups.com>

https://drive.google.com/open?id=0Bxs_ao6uuBDUSEc5S3U3Nko5ZjA


A. i am not sure whether parameter start from 0 or 1
n[0] or n[1] in compute function

B. it run a very long time and nothing to see in amazon linux instance
there is no python program in top command

C. in distributed programming web site, there is no authentication method
using private key in python program,  how do distributed program access
the nodes in amazon cloud when run python program below in window locally?

import random, dispy

def compute(n): # executed on nodes
    import random, time, socket
    name = socket.gethostname()
    cur_best = 1
    for ii in range(n[0],n[0]):
        for jj in range(n[1],n[1]):
            for kk in range(n[2],n[3],100):     
                dispy_provisional_result((name, r))
                cur_best = r
                time.sleep(0.1)
                # final result
    return (name, cur_best)

def job_callback(job): # executed at the client
    if job.status == dispy.DispyJob.ProvisionalResult:
        #if job.result[1] < 0.005:
            # acceptable result; terminate jobs
            print('%s computed: %s %s %s %s' % (job.result[0], job.result[1], job.result[2], job.result[3], job.result[4]))
            # 'jobs' and 'cluster' are created in '__main__' below
            for j in jobs:
                if j.status in [dispy.DispyJob.Created, dispy.DispyJob.Running,
                                dispy.DispyJob.ProvisionalResult]:
                    cluster.cancel(j)

if __name__ == '__main__':
    #cluster = dispy.JobCluster(compute, callback=job_callback)
    cluster = dispy.JobCluster(compute, nodes=['ec2-100-162-137-237.us-west-2.compute.amazonaws.com'], callback=job_callback)
    jobs = []
    prevk = 1
    count = 0
    for ii in range(1,2):
        for jj in range(1,2000):
            for kk in range(1,2000,100):
                if ii < jj and jj < kk:
                    job = cluster.submit([ii,jj,prevk,kk])
                    prevk = kk
                    if job is None:
                        print('creating job %s failed!' % n)
                        continue
                    job.id = count
                    count = count + 1
                    jobs.append(job)
    cluster.wait()
    cluster.print_status()
    cluster.close()


From steve+python at pearwood.info  Tue Nov  1 03:58:09 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Tue, 01 Nov 2016 18:58:09 +1100
Subject: how to debug this distributed program?
References: <9874b2d1-25b1-4d73-a96f-906f9584f750@googlegroups.com>
Message-ID: <58184b11$0$1604$c3e8da3$5496439d@news.astraweb.com>

On Tue, 1 Nov 2016 06:40 pm, meInvent bbird wrote:

> how to debug this distributed program?


The same way you would debug any other program.

http://sscce.org/





-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From jobmattcon at gmail.com  Tue Nov  1 04:14:19 2016
From: jobmattcon at gmail.com (meInvent bbird)
Date: Tue, 1 Nov 2016 01:14:19 -0700 (PDT)
Subject: how to debug this distributed program?
In-Reply-To: <58184b11$0$1604$c3e8da3$5496439d@news.astraweb.com>
References: <9874b2d1-25b1-4d73-a96f-906f9584f750@googlegroups.com>
 <58184b11$0$1604$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <c06ec84f-8dbe-4eec-9fc4-56ebd086b4f5@googlegroups.com>

i follow documentation and add keyfile to indicate the path of private key
but still no response from program,

what is the correct way to do?

cluster = dispy.JobCluster(compute, nodes=['ec2-35-162-137-237.us-west-2.compute.amazonaws.com'], callback=job_callback, keyfile=r"C:\Users\martlee2\Downloads\datacenterusekey.ppk")



On Tuesday, November 1, 2016 at 3:58:24 PM UTC+8, Steve D'Aprano wrote:
> On Tue, 1 Nov 2016 06:40 pm, meInvent bbird wrote:
> 
> > how to debug this distributed program?
> 
> 
> The same way you would debug any other program.
> 
> http://sscce.org/
> 
> 
> 
> 
> 
> -- 
> Steve
> ?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
> enough, things got worse.



From pavel.velikhov at gmail.com  Tue Nov  1 04:56:02 2016
From: pavel.velikhov at gmail.com (Pavel Velikhov)
Date: Tue, 1 Nov 2016 01:56:02 -0700 (PDT)
Subject: Announcement: PythonQL - An integrated query language for Python
Message-ID: <322f290d-3ccd-4e80-9ac6-862ee387e3a9@googlegroups.com>

Hi Folks,

 We have released PythonQL, a query language extension to Python (we have extended Python?s comprehensions with a full-fledged query language,
drawing from the useful features of SQL, XQuery and JSONiq). Take a look at the project here: http://www.pythonql.org and lets us know what you think!

 The way PythonQL currently works is you mark PythonQL files with a special encoding and the system runs a preprocessor for all such files (this is similar to how pyxl project has been done). We have an interactive interpreter and Jupyter support planned.

Best regards!
PythonQL team


From __peter__ at web.de  Tue Nov  1 05:49:45 2016
From: __peter__ at web.de (Peter Otten)
Date: Tue, 01 Nov 2016 10:49:45 +0100
Subject: Announcement: PythonQL - An integrated query language for Python
References: <322f290d-3ccd-4e80-9ac6-862ee387e3a9@googlegroups.com>
Message-ID: <nv9ofp$aps$1@blaine.gmane.org>

Pavel Velikhov wrote:

>  We have released PythonQL, a query language extension to Python (we have
>  extended Python?s comprehensions with a full-fledged query language,
> drawing from the useful features of SQL, XQuery and JSONiq). Take a look
> at the project here: http://www.pythonql.org and lets us know what you
> think!

I would really like Python to get seamless integration of SQL, so I applaud 
your effort!

The demo on your web page is too noisy for my taste, so I went ahead and 
installed the python3 version in a virtual env.

My first attempt failed because of a missing ply; maybe you can fix that.
Now off to the tutorial...
 
>  The way PythonQL currently works is you mark PythonQL files with a
>  special encoding and the system runs a preprocessor for all such files
>  (this is similar to how pyxl project has been done). We have an
>  interactive interpreter and Jupyter support planned.
> 
> Best regards!
> PythonQL team




From and.damore at gmail.com  Tue Nov  1 06:34:46 2016
From: and.damore at gmail.com (Andrea D'Amore)
Date: Tue, 1 Nov 2016 11:34:46 +0100
Subject: Need help with coding a function in Python
In-Reply-To: <01d2a11b-922c-4a07-a203-58238b2a7a82@googlegroups.com>
References: <01d2a11b-922c-4a07-a203-58238b2a7a82@googlegroups.com>
Message-ID: <CAMonFrQGdML0U9EUOom6gG4WcS-_mkw4STRLOdvqsVKJyoVYVg@mail.gmail.com>

On 31 October 2016 at 23:09,  <devers.meetthebadger.jason at gmail.com> wrote:
> http://imgur.com/a/rfGhK#iVLQKSW
> How do I code a function that returns a list of the first n elements
> of the sequence defined in the link? I have no idea!!!!!

For those who didn't open the page (that you should have linked at
least as direct link to the image rather than to the javascript based
frontend of imgur) here's the description: there's a mathematical
sequence where

    a_0 = 0
    a_n = a_{n-1} - n    if a_{n-1} is positive and not already in the sequence
          a_{n-1} + n    otherwise

so it's based off a simple sequence of kind   a_n = a_{n-1} + n   with
a conditional that brings the value back at times.


> So far this is my best shot at it (the problem with it is that the n that
> i'm subtracting or adding in the if/else part does not represent the
> element's position, but just the n that I am plugging into the function):

Since I've been lately trying to tackle small problems in the most
idiomatic way I can (much to Raymond Hettinger's enjoyable talks'
fault) I had my attempt at it.
You tried to build a list and return it while I make a function whose
return value is then iterated upon:


    def sequence(n):
        """Returns a generator for the mathematical sequence

            a_0 = 0
            a_n = a_{n-1} - n    if a_{n-1} is positive and not
already in the sequence
                  a_{n-1} + n    otherwise
        """

        value, past = 0, {}

        for c in range(n):
            t = value - c
            value = t if (t > 0 and t not in past) else (value + c)

            past[value] = True

            yield value



I'm not sure this can be made without the past state in the closure
since the sequence conditional rule depends on it.

Any hints on how to make it better are welcome.


-- 
Andrea


From kaiser.yann at gmail.com  Tue Nov  1 09:04:16 2016
From: kaiser.yann at gmail.com (Yann Kaiser)
Date: Tue, 01 Nov 2016 13:04:16 +0000
Subject: how to compile this code
In-Reply-To: <eae2aa23-0be0-4322-9245-609ec85c3859@googlegroups.com>
References: <eae2aa23-0be0-4322-9245-609ec85c3859@googlegroups.com>
Message-ID: <CANUJvPWSNt+2O2iBK9=ZcBK-7mKyxAP2MjP5c_whKOpecXtwWA@mail.gmail.com>

You want to replace the `Add` ast with a `Call` ast rather than just
calling your function.

Something like:

    if isinstance(node.op, ast.Add):
        return ast.Call(some_ast_expression_that_will_evaluate_to_op,
[node.left, node.right], [])

You'll have to replace some_ast_expression_... to something like
Name(id="op2", ctx=ast.Load()) and inject op2 in the globals when you call
`exec`.

On Tue, Nov 1, 2016, 06:36 meInvent bbird <jobmattcon at gmail.com> wrote:

> would like to change Add operator to custom function op2 in solve function
> and then this solve([x*y - 1, x + 2], x, y)
> during solve, the parameters also change Add to custom function op2
>
> import ast
> from __future__ import division
> from sympy import *
> x, y, z, t = symbols('x y z t')
> k, m, n = symbols('k m n', integer=True)
> f, g, h = symbols('f g h', cls=Function)
> import inspect
>
> def op2(a,b):
>     return a*b+a
>
> class ChangeAddToMultiply(ast.NodeTransformer):
>     """Wraps all integers in a call to Integer()"""
>     def visit_BinOp(self, node):
>         print(dir(node))
>         print(dir(node.left))
>         if isinstance(node.op, ast.Add):
>             node.op = op2(node.left, node.right)
>         return node
>
> code = inspect.getsourcelines(solve)
> tree = ast.parse(code)
> tree = ChangeAddToMultiply().visit(tree)
> ast.fix_missing_locations(tree)
> co = compile(tree, '<ast>', "exec")
>
> exec(code)
> exec(co)
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
Yann Kaiser
kaiser.yann at gmail.com
yann.kaiser at efrei.net
+33 6 51 64 01 89
https://github.com/epsy


From grant.b.edwards at gmail.com  Tue Nov  1 09:42:03 2016
From: grant.b.edwards at gmail.com (Grant Edwards)
Date: Tue, 1 Nov 2016 13:42:03 +0000 (UTC)
Subject: Call a shell command from Python
References: <bOWdnTgDbsjFW4vFnZ2dnUU7-LPNnZ2d@giganews.com>
 <85r36x9lgi.fsf@benfinney.id.au>
 <mailman.19.1477889083.17234.python-list@python.org>
 <It2dnWjna7N5xIrFnZ2dnUU7-RednZ2d@giganews.com>
 <85mvhk9fbv.fsf@benfinney.id.au>
 <mailman.31.1477962586.17234.python-list@python.org>
 <GdCdnZree_YZvIXFnZ2dnUU7-K-dnZ2d@giganews.com>
 <58182d94$0$22142$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <nva63a$1lb$1@blaine.gmane.org>

On 2016-11-01, Steve D'Aprano <steve+python at pearwood.info> wrote:
> On Tue, 1 Nov 2016 04:00 pm, Wildman wrote:
>
>> You are correct about that but, in this case grep never "sees" the '$'
>> sign.  Bash expands $USER to the actual user name beforehand.  If you
>> are on a Linux system, enter this into a terminal to illustrate:
>> 
>> sudo grep ^$USER\: /etc/shadow
>
> Bash is not involved here. Python is calling grep directly.

He's already been told this 6 times, and is willfully ignoring it.

Just give up.

-- 
Grant Edwards               grant.b.edwards        Yow! With YOU, I can be
                                  at               MYSELF ...  We don't NEED
                              gmail.com            Dan Rather ...



From best_lay at yahoo.com  Tue Nov  1 10:29:50 2016
From: best_lay at yahoo.com (Wildman)
Date: Tue, 01 Nov 2016 09:29:50 -0500
Subject: Call a shell command from Python
References: <bOWdnTgDbsjFW4vFnZ2dnUU7-LPNnZ2d@giganews.com>
 <85r36x9lgi.fsf@benfinney.id.au>
 <mailman.19.1477889083.17234.python-list@python.org>
 <It2dnWjna7N5xIrFnZ2dnUU7-RednZ2d@giganews.com>
 <85mvhk9fbv.fsf@benfinney.id.au>
 <mailman.31.1477962586.17234.python-list@python.org>
 <GdCdnZree_YZvIXFnZ2dnUU7-K-dnZ2d@giganews.com>
 <85h97rai4j.fsf@benfinney.id.au>
 <mailman.33.1477977839.17234.python-list@python.org>
Message-ID: <aq-dnc62NolDO4XFnZ2dnUU7-bGdnZ2d@giganews.com>

On Tue, 01 Nov 2016 16:23:08 +1100, Ben Finney wrote:

> Wildman via Python-list <python-list at python.org> writes:
> 
>> [?] in this case grep never "sees" the '$' sign. Bash expands $USER to
>> the actual user name beforehand.
> 
> I understand how Bash substitutes variables on the command line.
> 
> What I need to repeat, though: In this case, no, Bash doesn't do that
> because Bash isn't getting involved. Grep does in fact see the ?$? in
> the command argument, because nothing ever replaces it.

I understand now.  Thank you.

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From best_lay at yahoo.com  Tue Nov  1 10:33:17 2016
From: best_lay at yahoo.com (Wildman)
Date: Tue, 01 Nov 2016 09:33:17 -0500
Subject: Call a shell command from Python
References: <bOWdnTgDbsjFW4vFnZ2dnUU7-LPNnZ2d@giganews.com>
 <85r36x9lgi.fsf@benfinney.id.au>
 <mailman.19.1477889083.17234.python-list@python.org>
 <It2dnWjna7N5xIrFnZ2dnUU7-RednZ2d@giganews.com>
 <85mvhk9fbv.fsf@benfinney.id.au>
 <mailman.31.1477962586.17234.python-list@python.org>
 <GdCdnZree_YZvIXFnZ2dnUU7-K-dnZ2d@giganews.com>
 <58182d94$0$22142$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <aq-dncm2NokwOoXFnZ2dnUU7-bEAAAAA@giganews.com>

On Tue, 01 Nov 2016 16:52:18 +1100, Steve D'Aprano wrote:

> On Tue, 1 Nov 2016 04:00 pm, Wildman wrote:
> 
>> You are correct about that but, in this case grep never "sees" the '$'
>> sign.  Bash expands $USER to the actual user name beforehand.  If you
>> are on a Linux system, enter this into a terminal to illustrate:
>> 
>> sudo grep ^$USER\: /etc/shadow
> 
> Bash is not involved here. Python is calling grep directly.
> 
> 
> You don't have to believe us, you can test this yourself. Create a simple
> text file with a single line containing your username, and a simple Python
> script that calls grep as you have been:
> 
> 
> [steve at ando ~]$ echo $USER
> steve
> [steve at ando ~]$ cat foo.txt
> blah blah steve blah blah
> [steve at ando ~]$ cat greptest.py
> import subprocess
> cmdlist = ['grep', '$USER', 'foo.txt']
> p = subprocess.Popen(cmdlist, stdout=subprocess.PIPE,stderr=subprocess.PIPE)
> line, err = p.communicate()
> print err, line
> 
> [steve at ando ~]$ python2.7 greptest.py
> 
> [steve at ando ~]$
> 
> 
> 
> So there you have it: categorical proof that bash does not expand the
> string '$USER'. It cannot: bash is not involved in the subprocess call.
> Python calls grep directly.
> 
> If you want to expand the '$USER' string from Python, do it yourself:

I understand now.  That explains more clearly why my original
code did not work.  Thank you.

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From best_lay at yahoo.com  Tue Nov  1 10:34:40 2016
From: best_lay at yahoo.com (Wildman)
Date: Tue, 01 Nov 2016 09:34:40 -0500
Subject: Call a shell command from Python
References: <bOWdnTgDbsjFW4vFnZ2dnUU7-LPNnZ2d@giganews.com>
 <85r36x9lgi.fsf@benfinney.id.au>
 <mailman.19.1477889083.17234.python-list@python.org>
 <It2dnWjna7N5xIrFnZ2dnUU7-RednZ2d@giganews.com>
 <85mvhk9fbv.fsf@benfinney.id.au>
 <mailman.31.1477962586.17234.python-list@python.org>
 <GdCdnZree_YZvIXFnZ2dnUU7-K-dnZ2d@giganews.com>
 <58182d94$0$22142$c3e8da3$5496439d@news.astraweb.com>
 <nva63a$1lb$1@blaine.gmane.org>
 <mailman.38.1478007789.17234.python-list@python.org>
Message-ID: <aq-dnci2NomdNYXFnZ2dnUU7-bGdnZ2d@giganews.com>

On Tue, 01 Nov 2016 13:42:03 +0000, Grant Edwards wrote:

> On 2016-11-01, Steve D'Aprano <steve+python at pearwood.info> wrote:
>> On Tue, 1 Nov 2016 04:00 pm, Wildman wrote:
>>
>>> You are correct about that but, in this case grep never "sees" the '$'
>>> sign.  Bash expands $USER to the actual user name beforehand.  If you
>>> are on a Linux system, enter this into a terminal to illustrate:
>>> 
>>> sudo grep ^$USER\: /etc/shadow
>>
>> Bash is not involved here. Python is calling grep directly.
> 
> He's already been told this 6 times, and is willfully ignoring it.
> 
> Just give up.

That was not my intention.  I was trying to understand that
which I did not understand.  Ever been there?

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From darcy at vex.net  Tue Nov  1 11:23:09 2016
From: darcy at vex.net (D'Arcy Cain)
Date: Tue, 1 Nov 2016 11:23:09 -0400
Subject: Call a shell command from Python
In-Reply-To: <85h97rai4j.fsf@benfinney.id.au>
References: <bOWdnTgDbsjFW4vFnZ2dnUU7-LPNnZ2d@giganews.com>
 <85r36x9lgi.fsf@benfinney.id.au>
 <mailman.19.1477889083.17234.python-list@python.org>
 <It2dnWjna7N5xIrFnZ2dnUU7-RednZ2d@giganews.com>
 <85mvhk9fbv.fsf@benfinney.id.au>
 <mailman.31.1477962586.17234.python-list@python.org>
 <GdCdnZree_YZvIXFnZ2dnUU7-K-dnZ2d@giganews.com>
 <85h97rai4j.fsf@benfinney.id.au>
Message-ID: <7e77920f-ebcd-4832-68ac-579815be2bc3@vex.net>

On 2016-11-01 01:23 AM, Ben Finney wrote:
> Wildman via Python-list <python-list at python.org> writes:
> So the way your script was invoked has no bearing on whether Bash will
> get involved in what your script does. Your script is *directly*
> invoking programs, and if you don't ask for a shell to be involved you
> won't get it.

In other words, the OP's script *is* bash in this scenario or a 
reasonable facsimile thereof.

The subject probably should have been "Emulating a shell in Python."  Of 
course, if the OP knew to use that subject he probably would have 
already had the answer.  :-)

-- 
D'Arcy J.M. Cain
System Administrator, Vex.Net
http://www.Vex.Net/ IM:darcy at Vex.Net
VoIP: sip:darcy at Vex.Net


From pavel.velikhov at gmail.com  Tue Nov  1 11:46:42 2016
From: pavel.velikhov at gmail.com (Pavel Velikhov)
Date: Tue, 1 Nov 2016 08:46:42 -0700 (PDT)
Subject: Announcement: PythonQL - An integrated query language for Python
In-Reply-To: <mailman.34.1477993823.17234.python-list@python.org>
References: <322f290d-3ccd-4e80-9ac6-862ee387e3a9@googlegroups.com>
 <nv9ofp$aps$1@blaine.gmane.org>
 <mailman.34.1477993823.17234.python-list@python.org>
Message-ID: <a7b68885-bf6b-4199-b91a-e62230b4456b@googlegroups.com>

On Tuesday, 1 November 2016 12:50:37 UTC+3, Peter Otten  wrote:
> Pavel Velikhov wrote:
> 
> >  We have released PythonQL, a query language extension to Python (we have
> >  extended Python?s comprehensions with a full-fledged query language,
> > drawing from the useful features of SQL, XQuery and JSONiq). Take a look
> > at the project here: http://www.pythonql.org and lets us know what you
> > think!
> 
> I would really like Python to get seamless integration of SQL, so I applaud 
> your effort!
> 
> The demo on your web page is too noisy for my taste, so I went ahead and 
> installed the python3 version in a virtual env.
> 

Great! Yes, we're hoping this will be useful to folks that like SQL and other query languages.

> My first attempt failed because of a missing ply; maybe you can fix that.
> Now off to the tutorial...

Oops, I have tested with virtual env and ply was installing just fine, wierd.
Any hints on why it didn't pick it up during the installation?
  
> >  The way PythonQL currently works is you mark PythonQL files with a
> >  special encoding and the system runs a preprocessor for all such files
> >  (this is similar to how pyxl project has been done). We have an
> >  interactive interpreter and Jupyter support planned.
> > 
> > Best regards!
> > PythonQL team



From __peter__ at web.de  Tue Nov  1 13:08:12 2016
From: __peter__ at web.de (Peter Otten)
Date: Tue, 01 Nov 2016 18:08:12 +0100
Subject: Announcement: PythonQL - An integrated query language for Python
References: <322f290d-3ccd-4e80-9ac6-862ee387e3a9@googlegroups.com>
 <nv9ofp$aps$1@blaine.gmane.org>
 <mailman.34.1477993823.17234.python-list@python.org>
 <a7b68885-bf6b-4199-b91a-e62230b4456b@googlegroups.com>
Message-ID: <nvai5t$s6f$1@blaine.gmane.org>

Pavel Velikhov wrote:

> On Tuesday, 1 November 2016 12:50:37 UTC+3, Peter Otten  wrote:
>> Pavel Velikhov wrote:
>> 
>> >  We have released PythonQL, a query language extension to Python (we
>> >  have extended Python?s comprehensions with a full-fledged query
>> >  language,
>> > drawing from the useful features of SQL, XQuery and JSONiq). Take a
>> > look at the project here: http://www.pythonql.org and lets us know what
>> > you think!
>> 
>> I would really like Python to get seamless integration of SQL, so I
>> applaud your effort!
>> 
>> The demo on your web page is too noisy for my taste, so I went ahead and
>> installed the python3 version in a virtual env.
>> 
> 
> Great! Yes, we're hoping this will be useful to folks that like SQL and
> other query languages.
> 
>> My first attempt failed because of a missing ply; maybe you can fix that.
>> Now off to the tutorial...
> 
> Oops, I have tested with virtual env and ply was installing just fine,
> wierd. Any hints on why it didn't pick it up during the installation?

I don't know enough about pip to make sense of it, but here's what I see:

$ virtualenv -p python3 tmp_pyql
Running virtualenv with interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in tmp_pyql/bin/python3
Also creating executable in tmp_pyql/bin/python
Installing setuptools, pip...done.
$ cd tmp_pyql/
$ . bin/activate


If at this point I first run

(tmp_pyql)$ run pip install ply

installing pythonql3 will succeed. Otherwise:


(tmp_pyql)$ pip install pythonql3
Downloading/unpacking pythonql3
  Downloading pythonql3-0.9.43.tar.gz (41kB): 41kB downloaded
  Running setup.py (path:/home/peter/tmp_pyql/build/pythonql3/setup.py) 
egg_info for package pythonql3
    
Downloading/unpacking ply>=3.9 (from pythonql3)
  Downloading ply-3.9.tar.gz (150kB): 150kB downloaded
  Running setup.py (path:/home/peter/tmp_pyql/build/ply/setup.py) egg_info 
for package ply
    
    warning: no previously-included files matching '*.pyc' found anywhere in 
distribution
Installing collected packages: pythonql3, ply
  Running setup.py install for pythonql3
    
    Running post install task
  Running setup.py install for ply
    Failed to import the site module
    Traceback (most recent call last):
      File "/home/peter/tmp_pyql/lib/python3.4/site.py", line 703, in 
<module>
        main()
      File "/home/peter/tmp_pyql/lib/python3.4/site.py", line 683, in main
        paths_in_sys = addsitepackages(paths_in_sys)
      File "/home/peter/tmp_pyql/lib/python3.4/site.py", line 282, in 
addsitepackages
        addsitedir(sitedir, known_paths)
      File "/home/peter/tmp_pyql/lib/python3.4/site.py", line 204, in 
addsitedir
        addpackage(sitedir, name, known_paths)
      File "/home/peter/tmp_pyql/lib/python3.4/site.py", line 173, in 
addpackage
        exec(line)
      File "<string>", line 1, in <module>
      File "/home/peter/tmp_pyql/lib/python3.4/site-
packages/pythonql/codec/register.py", line 5, in <module>
        from pythonql.parser.Preprocessor import makeProgramFromString
      File "/home/peter/tmp_pyql/lib/python3.4/site-
packages/pythonql/parser/Preprocessor.py", line 3, in <module>
        from pythonql.parser.PythonQLParser import Parser, Node, 
print_program
      File "/home/peter/tmp_pyql/lib/python3.4/site-
packages/pythonql/parser/PythonQLParser.py", line 1, in <module>
        import ply.yacc as yacc
    ImportError: No module named 'ply'
    Complete output from command /home/peter/tmp_pyql/bin/python3 -c "import 
setuptools, 
tokenize;__file__='/home/peter/tmp_pyql/build/ply/setup.py';exec(compile(getattr(tokenize, 
'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" 
install --record /tmp/pip-3yg_3xh1-record/install-record.txt --single-
version-externally-managed --compile --install-headers 
/home/peter/tmp_pyql/include/site/python3.4:
    Failed to import the site module

Traceback (most recent call last):

  File "/home/peter/tmp_pyql/lib/python3.4/site.py", line 703, in <module>

    main()

  File "/home/peter/tmp_pyql/lib/python3.4/site.py", line 683, in main

    paths_in_sys = addsitepackages(paths_in_sys)

  File "/home/peter/tmp_pyql/lib/python3.4/site.py", line 282, in 
addsitepackages

    addsitedir(sitedir, known_paths)

  File "/home/peter/tmp_pyql/lib/python3.4/site.py", line 204, in addsitedir

    addpackage(sitedir, name, known_paths)

  File "/home/peter/tmp_pyql/lib/python3.4/site.py", line 173, in addpackage

    exec(line)

  File "<string>", line 1, in <module>

  File "/home/peter/tmp_pyql/lib/python3.4/site-
packages/pythonql/codec/register.py", line 5, in <module>

    from pythonql.parser.Preprocessor import makeProgramFromString

  File "/home/peter/tmp_pyql/lib/python3.4/site-
packages/pythonql/parser/Preprocessor.py", line 3, in <module>

    from pythonql.parser.PythonQLParser import Parser, Node, print_program

  File "/home/peter/tmp_pyql/lib/python3.4/site-
packages/pythonql/parser/PythonQLParser.py", line 1, in <module>

    import ply.yacc as yacc

ImportError: No module named 'ply'

----------------------------------------
Cleaning up...
Command /home/peter/tmp_pyql/bin/python3 -c "import setuptools, 
tokenize;__file__='/home/peter/tmp_pyql/build/ply/setup.py';exec(compile(getattr(tokenize, 
'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" 
install --record /tmp/pip-3yg_3xh1-record/install-record.txt --single-
version-externally-managed --compile --install-headers 
/home/peter/tmp_pyql/include/site/python3.4 failed with error code 1 in 
/home/peter/tmp_pyql/build/ply
Storing debug log for failure in /home/peter/.pip/pip.log
(tmp_pyql)$ 


If I were to guess: for some reason ply is installed after pythonql3.



From torriem at gmail.com  Tue Nov  1 13:09:09 2016
From: torriem at gmail.com (Michael Torrie)
Date: Tue, 1 Nov 2016 11:09:09 -0600
Subject: Announcement: PythonQL - An integrated query language for Python
In-Reply-To: <322f290d-3ccd-4e80-9ac6-862ee387e3a9@googlegroups.com>
References: <322f290d-3ccd-4e80-9ac6-862ee387e3a9@googlegroups.com>
Message-ID: <f90c43c7-f2f0-92c4-6aeb-5e08a45b7978@gmail.com>

On 11/01/2016 02:56 AM, Pavel Velikhov wrote:
> Hi Folks,
> 
> We have released PythonQL, a query language extension to Python (we
> have extended Python?s comprehensions with a full-fledged query
> language, drawing from the useful features of SQL, XQuery and
> JSONiq). Take a look at the project here: http://www.pythonql.org and
> lets us know what you think!

Sounds interesting.  Hope you'll accept a wee bit of criticism of the
web page, though.  Why not put the examples clearly on the front page,
along with output?  Took me a while to find the examples, and it wasn't
obvious what to do with the examples matrix.  I'd prefer to just see
them straight away without too much digging.

I'm glad you had a link to your github repo right on the main page. I
found your Github markdown files to be far more informative than your
web page.


From pavel.velikhov at gmail.com  Tue Nov  1 14:46:01 2016
From: pavel.velikhov at gmail.com (Pavel Velikhov)
Date: Tue, 1 Nov 2016 11:46:01 -0700 (PDT)
Subject: Announcement: PythonQL - An integrated query language for Python
In-Reply-To: <mailman.41.1478020589.17234.python-list@python.org>
References: <322f290d-3ccd-4e80-9ac6-862ee387e3a9@googlegroups.com>
 <f90c43c7-f2f0-92c4-6aeb-5e08a45b7978@gmail.com>
 <mailman.41.1478020589.17234.python-list@python.org>
Message-ID: <4a4bc167-c268-45a8-ba1a-5df5779c687f@googlegroups.com>

On Tuesday, 1 November 2016 20:16:43 UTC+3, Michael Torrie  wrote:
> On 11/01/2016 02:56 AM, Pavel Velikhov wrote:
> > Hi Folks,
> > 
> > We have released PythonQL, a query language extension to Python (we
> > have extended Python?s comprehensions with a full-fledged query
> > language, drawing from the useful features of SQL, XQuery and
> > JSONiq). Take a look at the project here: http://www.pythonql.org and
> > lets us know what you think!
> 
> Sounds interesting.  Hope you'll accept a wee bit of criticism of the
> web page, though.  Why not put the examples clearly on the front page,
> along with output?  Took me a while to find the examples, and it wasn't
> obvious what to do with the examples matrix.  I'd prefer to just see
> them straight away without too much digging.
> 
> I'm glad you had a link to your github repo right on the main page. I
> found your Github markdown files to be far more informative than your
> web page.

Hi Michael!

Thanks for the feedback, will try to make the examples easier to find, definitely!
Not too happy with the site layout myself... Was it obvious that you can play around
with the examples - i.e. edit them and run modified versions?

Will keep on improving the documentation too!


From pavel.velikhov at gmail.com  Tue Nov  1 14:47:24 2016
From: pavel.velikhov at gmail.com (Pavel Velikhov)
Date: Tue, 1 Nov 2016 11:47:24 -0700 (PDT)
Subject: Announcement: PythonQL - An integrated query language for Python
In-Reply-To: <mailman.40.1478020139.17234.python-list@python.org>
References: <322f290d-3ccd-4e80-9ac6-862ee387e3a9@googlegroups.com>
 <nv9ofp$aps$1@blaine.gmane.org>
 <mailman.34.1477993823.17234.python-list@python.org>
 <a7b68885-bf6b-4199-b91a-e62230b4456b@googlegroups.com>
 <nvai5t$s6f$1@blaine.gmane.org>
 <mailman.40.1478020139.17234.python-list@python.org>
Message-ID: <0bbf29f5-2e6b-41b3-b6e6-0f756523a4a8@googlegroups.com>

On Tuesday, 1 November 2016 20:09:14 UTC+3, Peter Otten  wrote:
> Pavel Velikhov wrote:
> 
> > On Tuesday, 1 November 2016 12:50:37 UTC+3, Peter Otten  wrote:
> >> Pavel Velikhov wrote:
> >> 
> >> >  We have released PythonQL, a query language extension to Python (we
> >> >  have extended Python?s comprehensions with a full-fledged query
> >> >  language,
> >> > drawing from the useful features of SQL, XQuery and JSONiq). Take a
> >> > look at the project here: http://www.pythonql.org and lets us know what
> >> > you think!
> >> 
> >> I would really like Python to get seamless integration of SQL, so I
> >> applaud your effort!
> >> 
> >> The demo on your web page is too noisy for my taste, so I went ahead and
> >> installed the python3 version in a virtual env.
> >> 
> > 
> > Great! Yes, we're hoping this will be useful to folks that like SQL and
> > other query languages.
> > 
> >> My first attempt failed because of a missing ply; maybe you can fix that.
> >> Now off to the tutorial...
> > 
> > Oops, I have tested with virtual env and ply was installing just fine,
> > wierd. Any hints on why it didn't pick it up during the installation?
> 
> I don't know enough about pip to make sense of it, but here's what I see:
> 
> $ virtualenv -p python3 tmp_pyql
> Running virtualenv with interpreter /usr/bin/python3
> Using base prefix '/usr'
> New python executable in tmp_pyql/bin/python3
> Also creating executable in tmp_pyql/bin/python
> Installing setuptools, pip...done.
> $ cd tmp_pyql/
> $ . bin/activate
> 
> 
> If at this point I first run
> 
> (tmp_pyql)$ run pip install ply
> 
> installing pythonql3 will succeed. Otherwise:
> 
> 
> (tmp_pyql)$ pip install pythonql3
> Downloading/unpacking pythonql3
>   Downloading pythonql3-0.9.43.tar.gz (41kB): 41kB downloaded
>   Running setup.py (path:/home/peter/tmp_pyql/build/pythonql3/setup.py) 
> egg_info for package pythonql3
>     
> Downloading/unpacking ply>=3.9 (from pythonql3)
>   Downloading ply-3.9.tar.gz (150kB): 150kB downloaded
>   Running setup.py (path:/home/peter/tmp_pyql/build/ply/setup.py) egg_info 
> for package ply
>     
>     warning: no previously-included files matching '*.pyc' found anywhere in 
> distribution
> Installing collected packages: pythonql3, ply
>   Running setup.py install for pythonql3
>     
>     Running post install task
>   Running setup.py install for ply
>     Failed to import the site module
>     Traceback (most recent call last):
>       File "/home/peter/tmp_pyql/lib/python3.4/site.py", line 703, in 
> <module>
>         main()
>       File "/home/peter/tmp_pyql/lib/python3.4/site.py", line 683, in main
>         paths_in_sys = addsitepackages(paths_in_sys)
>       File "/home/peter/tmp_pyql/lib/python3.4/site.py", line 282, in 
> addsitepackages
>         addsitedir(sitedir, known_paths)
>       File "/home/peter/tmp_pyql/lib/python3.4/site.py", line 204, in 
> addsitedir
>         addpackage(sitedir, name, known_paths)
>       File "/home/peter/tmp_pyql/lib/python3.4/site.py", line 173, in 
> addpackage
>         exec(line)
>       File "<string>", line 1, in <module>
>       File "/home/peter/tmp_pyql/lib/python3.4/site-
> packages/pythonql/codec/register.py", line 5, in <module>
>         from pythonql.parser.Preprocessor import makeProgramFromString
>       File "/home/peter/tmp_pyql/lib/python3.4/site-
> packages/pythonql/parser/Preprocessor.py", line 3, in <module>
>         from pythonql.parser.PythonQLParser import Parser, Node, 
> print_program
>       File "/home/peter/tmp_pyql/lib/python3.4/site-
> packages/pythonql/parser/PythonQLParser.py", line 1, in <module>
>         import ply.yacc as yacc
>     ImportError: No module named 'ply'
>     Complete output from command /home/peter/tmp_pyql/bin/python3 -c "import 
> setuptools, 
> tokenize;__file__='/home/peter/tmp_pyql/build/ply/setup.py';exec(compile(getattr(tokenize, 
> 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" 
> install --record /tmp/pip-3yg_3xh1-record/install-record.txt --single-
> version-externally-managed --compile --install-headers 
> /home/peter/tmp_pyql/include/site/python3.4:
>     Failed to import the site module
> 
> Traceback (most recent call last):
> 
>   File "/home/peter/tmp_pyql/lib/python3.4/site.py", line 703, in <module>
> 
>     main()
> 
>   File "/home/peter/tmp_pyql/lib/python3.4/site.py", line 683, in main
> 
>     paths_in_sys = addsitepackages(paths_in_sys)
> 
>   File "/home/peter/tmp_pyql/lib/python3.4/site.py", line 282, in 
> addsitepackages
> 
>     addsitedir(sitedir, known_paths)
> 
>   File "/home/peter/tmp_pyql/lib/python3.4/site.py", line 204, in addsitedir
> 
>     addpackage(sitedir, name, known_paths)
> 
>   File "/home/peter/tmp_pyql/lib/python3.4/site.py", line 173, in addpackage
> 
>     exec(line)
> 
>   File "<string>", line 1, in <module>
> 
>   File "/home/peter/tmp_pyql/lib/python3.4/site-
> packages/pythonql/codec/register.py", line 5, in <module>
> 
>     from pythonql.parser.Preprocessor import makeProgramFromString
> 
>   File "/home/peter/tmp_pyql/lib/python3.4/site-
> packages/pythonql/parser/Preprocessor.py", line 3, in <module>
> 
>     from pythonql.parser.PythonQLParser import Parser, Node, print_program
> 
>   File "/home/peter/tmp_pyql/lib/python3.4/site-
> packages/pythonql/parser/PythonQLParser.py", line 1, in <module>
> 
>     import ply.yacc as yacc
> 
> ImportError: No module named 'ply'
> 
> ----------------------------------------
> Cleaning up...
> Command /home/peter/tmp_pyql/bin/python3 -c "import setuptools, 
> tokenize;__file__='/home/peter/tmp_pyql/build/ply/setup.py';exec(compile(getattr(tokenize, 
> 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" 
> install --record /tmp/pip-3yg_3xh1-record/install-record.txt --single-
> version-externally-managed --compile --install-headers 
> /home/peter/tmp_pyql/include/site/python3.4 failed with error code 1 in 
> /home/peter/tmp_pyql/build/ply
> Storing debug log for failure in /home/peter/.pip/pip.log
> (tmp_pyql)$ 
> 
> 
> If I were to guess: for some reason ply is installed after pythonql3.

Thank you Peter!!
Looking into this, really strange behaviour...


From best_lay at yahoo.com  Tue Nov  1 14:51:56 2016
From: best_lay at yahoo.com (Wildman)
Date: Tue, 01 Nov 2016 13:51:56 -0500
Subject: Call a shell command from Python
References: <bOWdnTgDbsjFW4vFnZ2dnUU7-LPNnZ2d@giganews.com>
 <85r36x9lgi.fsf@benfinney.id.au>
 <mailman.19.1477889083.17234.python-list@python.org>
 <It2dnWjna7N5xIrFnZ2dnUU7-RednZ2d@giganews.com>
 <85mvhk9fbv.fsf@benfinney.id.au>
 <mailman.31.1477962586.17234.python-list@python.org>
 <GdCdnZree_YZvIXFnZ2dnUU7-K-dnZ2d@giganews.com>
 <85h97rai4j.fsf@benfinney.id.au>
 <7e77920f-ebcd-4832-68ac-579815be2bc3@vex.net>
 <mailman.39.1478013798.17234.python-list@python.org>
Message-ID: <7tCdnSfQQ4rReYXFnZ2dnUU7-N2dnZ2d@giganews.com>

On Tue, 01 Nov 2016 11:23:09 -0400, D'Arcy Cain wrote:

> On 2016-11-01 01:23 AM, Ben Finney wrote:
>> Wildman via Python-list <python-list at python.org> writes:
>> So the way your script was invoked has no bearing on whether Bash will
>> get involved in what your script does. Your script is *directly*
>> invoking programs, and if you don't ask for a shell to be involved you
>> won't get it.
> 
> In other words, the OP's script *is* bash in this scenario or a 
> reasonable facsimile thereof.
> 
> The subject probably should have been "Emulating a shell in Python."  Of 
> course, if the OP knew to use that subject he probably would have 
> already had the answer.  :-)

You are absolutely correct.  And thank you for your understanding.

-- 
<Wildman> GNU/Linux user #557453
Keyboard not detected! Press any key to continue...


From demosthenesk at gmail.com  Tue Nov  1 17:21:47 2016
From: demosthenesk at gmail.com (Demosthenes Koptsis)
Date: Tue, 1 Nov 2016 23:21:47 +0200
Subject: [New Release] PyFTPD - an FTP server with GUI and CLI versions
Message-ID: <738bf73b-990a-3e88-cab9-b2c49935b8ff@gmail.com>

Hello,

i have just released the brand new FTP server based on pyftpdlib, named 
PyFTPD.

You can run it from CLI with PyFTPD-cli.py or if you like GUIs run the 
PyFTPD.py

It is written on PyQT4 and Python 2.7.12

More at https://github.com/demosthenesk/PyFTPD


Regards,

Dim



From torriem at gmail.com  Tue Nov  1 19:22:05 2016
From: torriem at gmail.com (Michael Torrie)
Date: Tue, 1 Nov 2016 17:22:05 -0600
Subject: Announcement: PythonQL - An integrated query language for Python
In-Reply-To: <4a4bc167-c268-45a8-ba1a-5df5779c687f@googlegroups.com>
References: <322f290d-3ccd-4e80-9ac6-862ee387e3a9@googlegroups.com>
 <f90c43c7-f2f0-92c4-6aeb-5e08a45b7978@gmail.com>
 <mailman.41.1478020589.17234.python-list@python.org>
 <4a4bc167-c268-45a8-ba1a-5df5779c687f@googlegroups.com>
Message-ID: <e0f972e2-25ba-8a06-019e-30eaf520cf4d@gmail.com>

On 11/01/2016 12:46 PM, Pavel Velikhov wrote:
> Thanks for the feedback, will try to make the examples easier to find, definitely!
> Not too happy with the site layout myself... Was it obvious that you can play around
> with the examples - i.e. edit them and run modified versions?

After I clicked on the example and then saw the output, it became clear.
 Not a bad feature.  Maybe showing the input in one area and the output
in another area of the same page would be clearer, and make it easier to
tweak things and see the results without having to go back to the
previous page to edit the code again.




From jobmattcon at gmail.com  Tue Nov  1 21:13:55 2016
From: jobmattcon at gmail.com (meInvent bbird)
Date: Tue, 1 Nov 2016 18:13:55 -0700 (PDT)
Subject: how to compile this code
In-Reply-To: <mailman.37.1478005469.17234.python-list@python.org>
References: <eae2aa23-0be0-4322-9245-609ec85c3859@googlegroups.com>
 <CANUJvPWSNt+2O2iBK9=ZcBK-7mKyxAP2MjP5c_whKOpecXtwWA@mail.gmail.com>
 <mailman.37.1478005469.17234.python-list@python.org>
Message-ID: <fc47cf16-1454-447b-8963-4112fd317451@googlegroups.com>

>>> class ChangeAddToMultiply(ast.NodeTransformer, ast2.NodeTransformer):
...     """Wraps all integers in a call to Integer()"""
...     def visit_BinOp(self, node):
...         print(dir(node))
...         print(dir(node.left))
...         if isinstance(node.op, ast.Add):
...             ast.Call(Name(id="op2", ctx=ast2.Load()), [node.left, node.right
], [])
...         return node
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'ast2' is not defined
>>>
>>> code = inspect.getsourcelines(solve)
>>> tree = ast.parse(code)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\ast.py", line 37, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
TypeError: expected a readable buffer object
>>> tree2 = ast.parse("def op2(a,b): return a*b+a")
>>> tree = ChangeAddToMultiply().visit(tree,tree2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'tree' is not defined
>>> ast.fix_missing_locations(tree)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'tree' is not defined
>>> co = compile(tree, '<ast>', "exec")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'tree' is not defined
>>>
>>> exec(code)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: exec: arg 1 must be a string, file, or code object
>>> exec(co)

*********************************************

def op2(a,b): 
    return a*b+a 

class ChangeAddToMultiply(ast.NodeTransformer, ast2.NodeTransformer): 
    """Wraps all integers in a call to Integer()""" 
    def visit_BinOp(self, node): 
        print(dir(node)) 
        print(dir(node.left)) 
        if isinstance(node.op, ast.Add): 
            ast.Call(Name(id="op2", ctx=ast2.Load()), [node.left, node.right], []) 
        return node 


code = inspect.getsourcelines(solve) 
tree = ast.parse(code) 
tree2 = ast.parse("def op2(a,b): return a*b+a")
tree = ChangeAddToMultiply().visit(tree,tree2) 
ast.fix_missing_locations(tree) 
co = compile(tree, '<ast>', "exec") 

exec(code) 
exec(co) 



On Tuesday, November 1, 2016 at 9:04:43 PM UTC+8, Yann Kaiser wrote:
> You want to replace the `Add` ast with a `Call` ast rather than just
> calling your function.
> 
> Something like:
> 
>     if isinstance(node.op, ast.Add):
>         return ast.Call(some_ast_expression_that_will_evaluate_to_op,
> [node.left, node.right], [])
> 
> You'll have to replace some_ast_expression_... to something like
> Name(id="op2", ctx=ast.Load()) and inject op2 in the globals when you call
> `exec`.
> 
> On Tue, Nov 1, 2016, 06:36 meInvent bbird <jobmattcon at gmail.com> wrote:
> 
> > would like to change Add operator to custom function op2 in solve function
> > and then this solve([x*y - 1, x + 2], x, y)
> > during solve, the parameters also change Add to custom function op2
> >
> > import ast
> > from __future__ import division
> > from sympy import *
> > x, y, z, t = symbols('x y z t')
> > k, m, n = symbols('k m n', integer=True)
> > f, g, h = symbols('f g h', cls=Function)
> > import inspect
> >
> > def op2(a,b):
> >     return a*b+a
> >
> > class ChangeAddToMultiply(ast.NodeTransformer):
> >     """Wraps all integers in a call to Integer()"""
> >     def visit_BinOp(self, node):
> >         print(dir(node))
> >         print(dir(node.left))
> >         if isinstance(node.op, ast.Add):
> >             node.op = op2(node.left, node.right)
> >         return node
> >
> > code = inspect.getsourcelines(solve)
> > tree = ast.parse(code)
> > tree = ChangeAddToMultiply().visit(tree)
> > ast.fix_missing_locations(tree)
> > co = compile(tree, '<ast>', "exec")
> >
> > exec(code)
> > exec(co)
> > --
> > https://mail.python.org/mailman/listinfo/python-list
> >
> -- 
> Yann Kaiser
> kaiser.yann at gmail.com
> yann.kaiser at efrei.net
> +33 6 51 64 01 89
> https://github.com/epsy



From the.armsx at gmail.com  Tue Nov  1 23:55:07 2016
From: the.armsx at gmail.com (the.armsx at gmail.com)
Date: Tue, 1 Nov 2016 20:55:07 -0700 (PDT)
Subject: cx_freeze_zipimporter_instance
Message-ID: <db5d2fc4-f1a7-41c7-b792-740822260f3c@googlegroups.com>

can anybody help me in this problem...


From nathan.ernst at gmail.com  Wed Nov  2 00:05:28 2016
From: nathan.ernst at gmail.com (Nathan Ernst)
Date: Tue, 1 Nov 2016 23:05:28 -0500
Subject: cx_freeze_zipimporter_instance
In-Reply-To: <db5d2fc4-f1a7-41c7-b792-740822260f3c@googlegroups.com>
References: <db5d2fc4-f1a7-41c7-b792-740822260f3c@googlegroups.com>
Message-ID: <CAPjBxg-ZRS3t-CWs=zCy48_-adnBz896eihhu3nd8k8iXkH9vQ@mail.gmail.com>

No, because you've not provided anything resembling a question or a
problem. Please provide a minimal example set of code that exposes the
problem you are encountering, and describe the problem you are having. And
note that we will not write code for you if it ends up looking like a
homework problem.

Regards

On Tue, Nov 1, 2016 at 10:55 PM, <the.armsx at gmail.com> wrote:

> can anybody help me in this problem...
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From steve+comp.lang.python at pearwood.info  Wed Nov  2 00:48:38 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Wed, 02 Nov 2016 15:48:38 +1100
Subject: cx_freeze_zipimporter_instance
References: <db5d2fc4-f1a7-41c7-b792-740822260f3c@googlegroups.com>
Message-ID: <58197027$0$2877$c3e8da3$76491128@news.astraweb.com>

On Wednesday 02 November 2016 14:55, the.armsx at gmail.com wrote:

> can anybody help me in this problem...

What problem?



-- 
Steven
git gets easier once you get the basic idea that branches are homeomorphic 
endofunctors mapping submanifolds of a Hilbert space.



From framstag at rus.uni-stuttgart.de  Wed Nov  2 03:27:05 2016
From: framstag at rus.uni-stuttgart.de (Ulli Horlacher)
Date: Wed, 2 Nov 2016 07:27:05 +0000 (UTC)
Subject: advanced SimpleHTTPServer?
Message-ID: <nvc4g9$l6i$1@news2.informatik.uni-stuttgart.de>

"python -m SimpleHTTPServer" is really cool :-)

But I need some more features:

- some kind of a chroot, to prevent file access higher then the base
  directory

- a directory listing mit date and size information

- an upload possibility

I could modify /usr/lib/python2.7/SimpleHTTPServer.py by myself, but maybe
someone already has implemented these features?

Use case: users with notebooks in a foreign (WiFi) LAN want to exchange
some files on-the-fly, quick-and-dirty. Using USB sticks is a no-go! :-)


-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher at tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/


From dieter at handshake.de  Wed Nov  2 04:27:08 2016
From: dieter at handshake.de (dieter)
Date: Wed, 02 Nov 2016 09:27:08 +0100
Subject: lxml and xpath(?)
References: <9ceddc7b-a3d0-479a-a009-29c4bdc15e72@googlegroups.com>
Message-ID: <87oa1ywalf.fsf@handshake.de>

Doug OLeary <dkoleary at olearycomputers.com> writes:
> ...
> Any hints/tips/suggestions greatly appreciated especially with complete noob tutorials for xpath.

You can certainly do it with "XPath" (look for the "following-sibling" axis).

You can also use Python (with "lxml"). If you have an element "e", then
"e.getnext()" gives you the sibling following "e" (or `None`).



From chris at simplistix.co.uk  Wed Nov  2 04:32:33 2016
From: chris at simplistix.co.uk (Chris Withers)
Date: Wed, 2 Nov 2016 08:32:33 +0000
Subject: testfixtures 4.13.0 Released!
Message-ID: <72b90472-850b-3022-2dd4-a7f50f3635ad@simplistix.co.uk>

Hi All,

I'm pleased to announce the release of testfixtures 4.13.0 featuring the 
following:

- Add support to compare() for ignoring broken __eq__
   implementations, such as that found in the Django ORM!

The package is on PyPI and a full list of all the links to docs, issue 
trackers and the like can be found here:

https://github.com/Simplistix/testfixtures

Any questions, please do ask on the Testing in Python list or on the 
Simplistix open source mailing list...

cheers,

Chris


From and.damore at gmail.com  Wed Nov  2 08:26:04 2016
From: and.damore at gmail.com (Andrea D'Amore)
Date: Wed, 2 Nov 2016 13:26:04 +0100
Subject: advanced SimpleHTTPServer?
In-Reply-To: <nvc4g9$l6i$1@news2.informatik.uni-stuttgart.de>
References: <nvc4g9$l6i$1@news2.informatik.uni-stuttgart.de>
Message-ID: <CAMonFrT=Twgi=ELfwsCHijwDYnEKsBXtVsjCF-_mSDFdbaT5Bw@mail.gmail.com>

On 2 November 2016 at 08:27, Ulli Horlacher
<framstag at rus.uni-stuttgart.de> wrote:
> "python -m SimpleHTTPServer" is really cool :-)

> - some kind of a chroot, to prevent file access higher then the base
>   directory

Shouldn't that be done by chrooting the python process in the calling
environment?


-- 
Andrea


From stestagg at gmail.com  Wed Nov  2 09:46:25 2016
From: stestagg at gmail.com (stestagg at gmail.com)
Date: Wed, 2 Nov 2016 06:46:25 -0700 (PDT)
Subject: constructor classmethods
Message-ID: <7f7156ed-e396-460e-8eb1-fb6b17bab069@googlegroups.com>

Hi

I was hoping to canvas opinion on using classmethods as constructors over __init__.

We've got a colleague who is very keen that __init__ methods don't contain any logic/implementation at all, and if there is any, then it should be moved to a create() classmethod.

As a concrete example, one change proposed was to go from this:

```
def __init__(self, ...):
    self.queue = Queue.Queue()

to this:

def __init__(self, queue):
    self.queue = queue

@classmethod
def create(cls, ...):
    return cls(Queue.Queue())

```

The rationale is that it decouples the class from the queue implementation (no evidence or suggestion that we would actually ever want to change impl in this case), and makes testing easier.

I get the underlying principal, and it's one that a strict OOp approach would suggest, but my gut feeling is that this is not a pythonic approach at all.

What do people feel about this?

Thanks

Steve


From rosuav at gmail.com  Wed Nov  2 10:03:35 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 3 Nov 2016 01:03:35 +1100
Subject: constructor classmethods
In-Reply-To: <7f7156ed-e396-460e-8eb1-fb6b17bab069@googlegroups.com>
References: <7f7156ed-e396-460e-8eb1-fb6b17bab069@googlegroups.com>
Message-ID: <CAPTjJmpR8ZZ4XNdTkFqu6rS98W5Qitpu8ChzwOGsijPgpjaPHw@mail.gmail.com>

On Thu, Nov 3, 2016 at 12:46 AM,  <stestagg at gmail.com> wrote:
> We've got a colleague who is very keen that __init__ methods don't contain any logic/implementation at all, and if there is any, then it should be moved to a create() classmethod.
>
> I get the underlying principal, and it's one that a strict OOp approach would suggest, but my gut feeling is that this is not a pythonic approach at all.
>
> What do people feel about this?

Will you ever call the normal constructor? If the class depends on a
queue, as in your example, what happens if you call it with the same
queue as another instance is using, or something other than a queue,
etc, etc, etc? What this model does is introduce a whole lot of new
bug possibilities, for no particular reason.

I'll split your rationale into two:

> The rationale is that it decouples the class from the queue implementation (no evidence or suggestion that we would actually ever want to change impl in this case),
>

Then don't bother. YAGNI. No benefit unless you actually know you want this.

> and makes testing easier.
>

This is an important consideration, but shouldn't infect the primary
API. Design __init__ for normal usage, and if you need a mock queue
for testing, design an alternative way to construct one with a mock
queue. One way is to subclass your class to make a "mocked_class",
which then uses a mocked queue and whatever else it needs; another way
would be a static "_create_for_testing" function. Or you could just
create one as normal, then reach inside and replace its queue, but
that seems a tad risky (at very least, I'd make a method
"_replace_queue_for_testing" in the main class, so you have a chance
of noticing the problem as you're editing).

Ideally, testing should be done on the exact same code that you're
running, because otherwise you risk proving nothing. Minimize the code
used specifically for testing, and try to isolate it where possible.
And only warp the code when you *know* you need to - don't craft your
class's API around "well, we *might* need this for testing".

ChrisA


From breamoreboy at gmail.com  Wed Nov  2 10:06:20 2016
From: breamoreboy at gmail.com (breamoreboy at gmail.com)
Date: Wed, 2 Nov 2016 07:06:20 -0700 (PDT)
Subject: constructor classmethods
In-Reply-To: <7f7156ed-e396-460e-8eb1-fb6b17bab069@googlegroups.com>
References: <7f7156ed-e396-460e-8eb1-fb6b17bab069@googlegroups.com>
Message-ID: <bf11d3ae-556c-4dc8-a74d-0110d7877f09@googlegroups.com>

On Wednesday, November 2, 2016 at 1:47:00 PM UTC, stes... at gmail.com wrote:
> Hi
> 
> I was hoping to canvas opinion on using classmethods as constructors over __init__.
> 
> We've got a colleague who is very keen that __init__ methods don't contain any logic/implementation at all, and if there is any, then it should be moved to a create() classmethod.
> 
> As a concrete example, one change proposed was to go from this:
> 
> ```
> def __init__(self, ...):
>     self.queue = Queue.Queue()
> 
> to this:
> 
> def __init__(self, queue):
>     self.queue = queue
> 
> @classmethod
> def create(cls, ...):
>     return cls(Queue.Queue())
> 
> ```
> 
> The rationale is that it decouples the class from the queue implementation (no evidence or suggestion that we would actually ever want to change impl in this case), and makes testing easier.
> 
> I get the underlying principal, and it's one that a strict OOp approach would suggest, but my gut feeling is that this is not a pythonic approach at all.
> 
> What do people feel about this?
> 
> Thanks
> 
> Steve

I suggest that you read this https://docs.python.org/3/reference/datamodel.html#object.__new__ before you do anything.

Kindest regards.

Mark Lawrence.


From tjreedy at udel.edu  Wed Nov  2 10:34:07 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 2 Nov 2016 10:34:07 -0400
Subject: constructor classmethods
In-Reply-To: <7f7156ed-e396-460e-8eb1-fb6b17bab069@googlegroups.com>
References: <7f7156ed-e396-460e-8eb1-fb6b17bab069@googlegroups.com>
Message-ID: <nvctgt$jn6$1@blaine.gmane.org>

On 11/2/2016 9:46 AM, stestagg at gmail.com wrote:
> Hi
>
> I was hoping to canvas opinion on using classmethods as constructors over __init__.
>
> We've got a colleague who is very keen that __init__ methods don't contain any logic/implementation at all, and if there is any, then it should be moved to a create() classmethod.
>
> As a concrete example, one change proposed was to go from this:
>
> ```
> def __init__(self, ...):
>     self.queue = Queue.Queue()
>
> to this:
>
> def __init__(self, queue):
>     self.queue = queue
>
> @classmethod
> def create(cls, ...):
>     return cls(Queue.Queue())

Even with the above fixed to properly pass on the ... part, UGH!  It 
violate Occam's Razor and is harder to read. thi

> The rationale is that it decouples the class from the queue implementation (no evidence or suggestion that we would actually ever want to change impl in this case),

It just moves the dependency to a different method.

 > and makes testing easier.

If one wants to replace the the Queue with a MockQueue for testing, amd 
the class is named 'C', one can do

c = C(...)
c.queue = MockQueue()

or one can revise __init__ to allow for dependency injection

class C:
     def __init__(self, ..., queue=None):
         if queue is None:
             queue = Queue.Queue

and the test can have

c = C(..., queue=MockQueue)

> I get the underlying principal, and it's one that a strict OOp approach would suggest, but my gut feeling is that this is not a pythonic approach at all.

Perhaps your collegue uses languages that do not have default arguments, 
which make his alternative unnecessary for Python.

> What do people feel about this?

AFAIK, the stdlib only has alternate constructor classmethods as 
specialized alternatives to __init__, not as replacements. Examples are 
dict.fromkeys, int.from_bytes, and other .fromxyz methods.  These are 
used when multiplexing the .__init__ API to include the alternate 
constructor args is considered worse than adding an extra method.

 >>> int(30.3)
30
 >>> int('30')
30
 >>> int(b'30')
30
 >>> int(b'30', 12)
36
 >>> int.from_bytes(b'30', 'big')
13104
 >>> int.from_bytes(b'30', 'little')
12339


-- 
Terry Jan Reedy



From ethan at stoneleaf.us  Wed Nov  2 10:40:12 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 02 Nov 2016 07:40:12 -0700
Subject: constructor classmethods
In-Reply-To: <7f7156ed-e396-460e-8eb1-fb6b17bab069@googlegroups.com>
References: <7f7156ed-e396-460e-8eb1-fb6b17bab069@googlegroups.com>
Message-ID: <5819FACC.8090508@stoneleaf.us>

On 11/02/2016 06:46 AM, stestagg at gmail.com wrote:

> I was hoping to canvas opinion on using classmethods as constructors over __init__.
>
> We've got a colleague who is very keen that __init__ methods don't contain any logic/implementation at all, and if there is any, then it should be moved to a create() classmethod.

So every Python programmer who is used to creating new instances with `ThatThing()` will have to learn that some of your classes need to be `ThatThing.create()` ?

Horrible idea.

The purpose of classmethods that create the same kind of object is to provide constructors that take different things -- for example:

class date(object):

     def __init__(self, year, month, day):
         self.year = year
         self.month = month
         self.day = day

     @classmethod
     def from_stardate(cls, sd):
         ad = sd - SOME_CONSTANT_VALUE
         year = ...
         month = ...
         day = ...
         return cls(year, month, day)

If the goal is to make testing easier, one way to do that is:

def __init__(self, ..., queue=None):
     if queue is None:
         queue = Queue.Queue()
     self.queue = queue

--
~Ethan~


From congloi1990 at gmail.com  Wed Nov  2 11:45:10 2016
From: congloi1990 at gmail.com (congloi1990 at gmail.com)
Date: Wed, 2 Nov 2016 08:45:10 -0700 (PDT)
Subject: Recursive generator in Python 3.5
In-Reply-To: <e7pumvFptduU1@mid.individual.net>
References: <bad29cc1-f1c6-4319-9520-fd059d70b464@googlegroups.com>
 <e7pumvFptduU1@mid.individual.net>
Message-ID: <17a0ffba-32ed-4af6-9f28-1c5346802cdd@googlegroups.com>

Thank you so much, this all clear for me now


From kwpolska at gmail.com  Wed Nov  2 12:15:51 2016
From: kwpolska at gmail.com (Chris Warrick)
Date: Wed, 2 Nov 2016 17:15:51 +0100
Subject: advanced SimpleHTTPServer?
In-Reply-To: <nvc4g9$l6i$1@news2.informatik.uni-stuttgart.de>
References: <nvc4g9$l6i$1@news2.informatik.uni-stuttgart.de>
Message-ID: <CAMw+j7Jes=L+BMQ2QymfxF7OWKTdOtdbCFzCRk1N6X7vB5unPw@mail.gmail.com>

On 2 November 2016 at 08:27, Ulli Horlacher
<framstag at rus.uni-stuttgart.de> wrote:
> "python -m SimpleHTTPServer" is really cool :-)
>
> But I need some more features:
>
> - some kind of a chroot, to prevent file access higher then the base
>   directory
>
> - a directory listing mit date and size information
>
> - an upload possibility
>
> I could modify /usr/lib/python2.7/SimpleHTTPServer.py by myself, but maybe
> someone already has implemented these features?
>
> Use case: users with notebooks in a foreign (WiFi) LAN want to exchange
> some files on-the-fly, quick-and-dirty. Using USB sticks is a no-go! :-)

SimpleHTTPServer is meant to be used for development and testing. It
should not be used for anything remotely serious for security and
speed reasons.

Instead, you should use a real web server, such as nginx or apache.
Those will do the first two properly, and the last one could be
handled by a simple-ish PHP script. Or a full-fledged app in Django or
Flask if you feel like it.

-- 
Chris Warrick <https://chriswarrick.com/>
PGP: 5EAAEA16


From esj at harvee.org  Wed Nov  2 12:56:25 2016
From: esj at harvee.org (Eric S. Johansson)
Date: Wed, 2 Nov 2016 12:56:25 -0400
Subject: advanced SimpleHTTPServer?
In-Reply-To: <CAMw+j7Jes=L+BMQ2QymfxF7OWKTdOtdbCFzCRk1N6X7vB5unPw@mail.gmail.com>
References: <nvc4g9$l6i$1@news2.informatik.uni-stuttgart.de>
 <CAMw+j7Jes=L+BMQ2QymfxF7OWKTdOtdbCFzCRk1N6X7vB5unPw@mail.gmail.com>
Message-ID: <6d205aab-dd5e-30bd-36dd-6c7fb3b0dfd3@harvee.org>



On 11/2/2016 12:15 PM, Chris Warrick wrote:
> SimpleHTTPServer is meant to be used for development and testing. It
> should not be used for anything remotely serious for security and
> speed reasons. 

Given that many people are trying to use SimpleHTTPServer for
"production" should teach us that the existing http servers are overly
complex or at the least, not meeting needs when folks are scaling down
services.  for example, I've written any number of "small" webapps with
a user base of 5-10. do I need apache+cgi/uwsgi+bottle? no.  I need
something like SimpleHTTPServer only more production ready.

serious Q: what would it take to make SimpleHTTPServer more secure.  It
is currently fast enough.  when I run bottle with it's derivative of 
SimpleHTTPServer, it is as fast as apache+uwsgi+bottle for 1-10 users
and much much easier to set up.

> Instead, you should use a real web server, such as nginx or apache.
> Those will do the first two properly, and the last one could be
> handled by a simple-ish PHP script. Or a full-fledged app in Django or
> Flask if you feel like it.
or bottle.  why does everyone seem to forget bottle? simple to set up,
easy to learn and useful for small to medium projects. 



From drekscott at gmail.com  Wed Nov  2 13:40:24 2016
From: drekscott at gmail.com (Dreyton Scott)
Date: Wed, 2 Nov 2016 10:40:24 -0700 (PDT)
Subject: Django Application Model Reference
Message-ID: <c6382323-fdbe-447d-9229-08c9b61e1ed6@googlegroups.com>

Hello. I am currently creating a notification django application that will need to be able to "hook" into another django application. What my app needs is a way to retrieve all model classes in the connected application. Is there a way to do this?


From drekscott at gmail.com  Wed Nov  2 13:44:27 2016
From: drekscott at gmail.com (Dreyton Scott)
Date: Wed, 2 Nov 2016 10:44:27 -0700 (PDT)
Subject: Django Application Model Reference
In-Reply-To: <c6382323-fdbe-447d-9229-08c9b61e1ed6@googlegroups.com>
References: <c6382323-fdbe-447d-9229-08c9b61e1ed6@googlegroups.com>
Message-ID: <e1e80737-4a33-4e5d-b53f-1188c0cc9225@googlegroups.com>

On Wednesday, November 2, 2016 at 1:40:35 PM UTC-4, Dreyton Scott wrote:
> Hello. I am currently creating a notification django application that will need to be able to "hook" into another django application. What my app needs is a way to retrieve all model classes in the connected application. Is there a way to do this?

In a way such that my app (a) is connected to app (b). Have (a) retrieve all model class instances in the connected app (b). Is that possible?


From kwpolska at gmail.com  Wed Nov  2 14:40:03 2016
From: kwpolska at gmail.com (Chris Warrick)
Date: Wed, 2 Nov 2016 19:40:03 +0100
Subject: advanced SimpleHTTPServer?
In-Reply-To: <6d205aab-dd5e-30bd-36dd-6c7fb3b0dfd3@harvee.org>
References: <nvc4g9$l6i$1@news2.informatik.uni-stuttgart.de>
 <CAMw+j7Jes=L+BMQ2QymfxF7OWKTdOtdbCFzCRk1N6X7vB5unPw@mail.gmail.com>
 <6d205aab-dd5e-30bd-36dd-6c7fb3b0dfd3@harvee.org>
Message-ID: <CAMw+j7JSZhLhtV_rgVSdPq5rKqsp_EgnwCgn-fSVnwQZtfYPCw@mail.gmail.com>

On 2 November 2016 at 17:56, Eric S. Johansson <esj at harvee.org> wrote:
>> Instead, you should use a real web server, such as nginx or apache.
>> Those will do the first two properly, and the last one could be
>> handled by a simple-ish PHP script. Or a full-fledged app in Django or
>> Flask if you feel like it.
> or bottle.  why does everyone seem to forget bottle? simple to set up,
> easy to learn and useful for small to medium projects.

The other frameworks are simple to set up, too. I?ve personally had
Unicode issues with Bottle, and it doesn?t even do sessions. Unlike
Flask, or of course Django.

Because, as the old saying goes, any sufficiently complicated Bottle
or Flask app contains an ad hoc, informally-specified, bug-ridden,
slow implementation of half of Django.

(In the form of various plugins to do databases, accounts, admin panels etc.)

-- 
Chris Warrick <https://chriswarrick.com/>
PGP: 5EAAEA16


From walters.justin01 at gmail.com  Wed Nov  2 14:54:11 2016
From: walters.justin01 at gmail.com (justin walters)
Date: Wed, 2 Nov 2016 11:54:11 -0700
Subject: advanced SimpleHTTPServer?
In-Reply-To: <CAMw+j7JSZhLhtV_rgVSdPq5rKqsp_EgnwCgn-fSVnwQZtfYPCw@mail.gmail.com>
References: <nvc4g9$l6i$1@news2.informatik.uni-stuttgart.de>
 <CAMw+j7Jes=L+BMQ2QymfxF7OWKTdOtdbCFzCRk1N6X7vB5unPw@mail.gmail.com>
 <6d205aab-dd5e-30bd-36dd-6c7fb3b0dfd3@harvee.org>
 <CAMw+j7JSZhLhtV_rgVSdPq5rKqsp_EgnwCgn-fSVnwQZtfYPCw@mail.gmail.com>
Message-ID: <CAO1D73H4v5TXGaqfcEvLdF=Dm4_ih5ZkZ2WmJz0BGuY8T+yYxA@mail.gmail.com>

On Wed, Nov 2, 2016 at 11:40 AM, Chris Warrick <kwpolska at gmail.com> wrote:

> The other frameworks are simple to set up, too. I?ve personally had
> Unicode issues with Bottle, and it doesn?t even do sessions. Unlike
> Flask, or of course Django.
>
> Because, as the old saying goes, any sufficiently complicated Bottle
> or Flask app contains an ad hoc, informally-specified, bug-ridden,
> slow implementation of half of Django.
>
> (In the form of various plugins to do databases, accounts, admin panels
> etc.)
>


I've found this to be the case whenever I see a Flask or Bottle project
that does anything
more than expose a data API. I think Flask is great for building small
applications or
microservices. Any more and you are essentially hacking together a bunch of
packages
that may or may not have been designed to work together.

I find that Django can suit almost any use case perfectly well. You can use
whatever part
of the framework you want and toss out/replace any part you don't want.
Plus, built in sessions
and authentication are fantastic time savers.


From graffatcolmingov at gmail.com  Wed Nov  2 15:49:57 2016
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Wed, 2 Nov 2016 14:49:57 -0500
Subject: Released: pip v9.0.0
In-Reply-To: <7CEFCE24-DFC3-4E9E-B7E6-ABB398853AB3@stufft.io>
References: <7CEFCE24-DFC3-4E9E-B7E6-ABB398853AB3@stufft.io>
Message-ID: <CAN-Kwu1kZuyYSN-CSooYjzVstqzSih2L9J4WkZ1epscfsRM5aA@mail.gmail.com>

Pick check? ;-)

On Nov 2, 2016 2:27 PM, "Donald Stufft" <donald at stufft.io> wrote:

> I?d like to announce the release of pip v9.0.
>
> This release features:
>
> * The 9.x series will be the last pip versions to support Python 2.6.
> * Support for Requires-Python (will require additional support in
> setuptools/PyPI) to allow releasing sdists that will be ignored by specific
> versions of Python (e.g. foobar 5.x doesn?t get downloaded on 2.6).
> * Ability to pass platform, Python version, implementation, etc into ``pip
> download`` to download wheels for other platforms.
> * Add a ``pick check`` command to check the state of installed
> dependencies.
> * Add new formats for ``pip list``, including a new columnar layout and a
> JSON format for ease of scripting.
>
> For a full list of changes, see https://pip.pypa.io/en/stable/news/ <
> https://pip.pypa.io/en/stable/news/>.
>
> ?
> Donald Stufft
>
>
>
> --
> https://mail.python.org/mailman/listinfo/python-announce-list
>
>         Support the Python Software Foundation:
>         http://www.python.org/psf/donations/
>


From esj at harvee.org  Wed Nov  2 15:52:40 2016
From: esj at harvee.org (Eric S. Johansson)
Date: Wed, 2 Nov 2016 15:52:40 -0400
Subject: advanced SimpleHTTPServer?
In-Reply-To: <CAMw+j7JSZhLhtV_rgVSdPq5rKqsp_EgnwCgn-fSVnwQZtfYPCw@mail.gmail.com>
References: <nvc4g9$l6i$1@news2.informatik.uni-stuttgart.de>
 <CAMw+j7Jes=L+BMQ2QymfxF7OWKTdOtdbCFzCRk1N6X7vB5unPw@mail.gmail.com>
 <6d205aab-dd5e-30bd-36dd-6c7fb3b0dfd3@harvee.org>
 <CAMw+j7JSZhLhtV_rgVSdPq5rKqsp_EgnwCgn-fSVnwQZtfYPCw@mail.gmail.com>
Message-ID: <4079c7a3-074f-e070-08be-e0168a72dfc4@harvee.org>



On 11/2/2016 2:40 PM, Chris Warrick wrote:
> Because, as the old saying goes, any sufficiently complicated Bottle
> or Flask app contains an ad hoc, informally-specified, bug-ridden,
> slow implementation of half of Django. (In the form of various plugins
> to do databases, accounts, admin panels etc.)

That's not a special attribute of bottle, flask or Django. Ad hoc,
informally specified, bug ridden slow implementations abound.  We focus
too much on scaling up and not enough on scaling down. We (designers) 
also have not properly addressed configuration complexity issues.

If I'm going do something once, if it cost me more than a couple of
hours to figure it out, it's too expensive in general but definitely if
I forget what I learned. That's why bottle/flask systems meet and need.
They're not too expensive to forget what you learned.

Django makes the cost of forgetting extremely expensive. I think of
using Django as career  rather than a toolbox.

So this brings me back to my question. What is missing in
SimpleHTTPServer to keep it from being secure enough?




From walters.justin01 at gmail.com  Wed Nov  2 16:59:46 2016
From: walters.justin01 at gmail.com (justin walters)
Date: Wed, 2 Nov 2016 13:59:46 -0700
Subject: advanced SimpleHTTPServer?
In-Reply-To: <4079c7a3-074f-e070-08be-e0168a72dfc4@harvee.org>
References: <nvc4g9$l6i$1@news2.informatik.uni-stuttgart.de>
 <CAMw+j7Jes=L+BMQ2QymfxF7OWKTdOtdbCFzCRk1N6X7vB5unPw@mail.gmail.com>
 <6d205aab-dd5e-30bd-36dd-6c7fb3b0dfd3@harvee.org>
 <CAMw+j7JSZhLhtV_rgVSdPq5rKqsp_EgnwCgn-fSVnwQZtfYPCw@mail.gmail.com>
 <4079c7a3-074f-e070-08be-e0168a72dfc4@harvee.org>
Message-ID: <CAO1D73H+=jmUxWBwTwG_62opJVpE-fZfZA10uDw+cHMPS6=8VA@mail.gmail.com>

On Wed, Nov 2, 2016 at 12:52 PM, Eric S. Johansson <esj at harvee.org> wrote:

> So this brings me back to my question. What is missing in
> SimpleHTTPServer to keep it from being secure enough?
>

There's no way to vet requests. You can't stop a request from accessing
anything
in the directory that SimpleHTTPServer is running in. I'm sure an
enterprising
individual could also probably access the shell session SimpleHTTPServer
is running in as well. I haven't looked into the internals very much, but
it is possible
an attacker could use eval() to run a Python script sent in a request body.
Not
sure about that last one. I'll have to try it myself and report back.


From steve+comp.lang.python at pearwood.info  Thu Nov  3 01:57:22 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Thu, 03 Nov 2016 16:57:22 +1100
Subject: constructor classmethods
References: <7f7156ed-e396-460e-8eb1-fb6b17bab069@googlegroups.com>
Message-ID: <581ad1c6$0$1513$c3e8da3$5496439d@news.astraweb.com>

On Thursday 03 November 2016 00:46, stestagg at gmail.com wrote:

> Hi
> 
> I was hoping to canvas opinion on using classmethods as constructors over
> __init__.
> 
> We've got a colleague who is very keen that __init__ methods don't contain
> any logic/implementation at all, and if there is any, then it should be moved
> to a create() classmethod.

Is your colleague expecting the caller to directly call this create() method? 
In other words, is create() part of the public API for the users of the class, 
as in the following:

    instance = MyClass.create(arg)

instead of this?

    instance = MyClass(arg)


I think that's just being unconventional for the sake of being unconventional. 
The standard Python API for classes is to call the class name, we write:

    x = float(arg)
    d = dict(mapping, key=value)
    obj = object()

rather than float.create(arg), dict.create(...) and object.create(). Go through 
the entire Python standard library, and you'll see that virtually ALL classes 
and types -- not just "primitives" like float and dict -- behave like this.

There are a handful of classes which provide an Alternate Constructor API, e.g. 
dict.fromkeys(sequence_of_keys). But that offers a distinct, separate API for 
creating dicts. Hence the *alternate* part.

The bottom line is, the standard way to call MyClass.create(arg) in Python is 
to leave out the ".create".

But perhaps create is meant to be part of the *internal* implementation, not an 
external API. In that case, things get a bit more interesting... see below.


> As a concrete example, one change proposed was to go from this:
> 
> ```
> def __init__(self, ...):
>     self.queue = Queue.Queue()
> 
> to this:
> 
> def __init__(self, queue):
>     self.queue = queue


That's the Dependency Injection design pattern. It's so trivially easy in 
Python that it hardly even deserves the name "Design Pattern". But trivial or 
not, it's very useful! You just have to write it the smart way.


> @classmethod
> def create(cls, ...):
>     return cls(Queue.Queue())
> 

This is not the smart way. This requires the caller to either create their own 
queue (occasionally useful, but not something you want to do all the time) or 
else call a non-standard constructor method.

The smart way of doing optional dependency injection is like this:


    def __init__(self, queue=None):
        if queue is None:
            queue = Queue.Queue()
        self.queue = queue


In more complex cases I can see the wisdom of writing a separate method:

    def __init__(self, widget=None):
        if widget is None:
            widget = self._build_widget(spam, eggs, cheese, tomato)
        self.widget = widget

    def _build_widget(self, a, b, c, d):
        ...
        return widget


Note that the create method, _build_widget(), is flagged as private by the 
leading underscore. Users should not call it (or they do so at their own risk); 
subclasses may override or replace the _build_widget method.

That may be what your colleague is thinking about. If so, then I can see that 
it *may* be a useful technique in some cases. But my guess is that he is over-
engineering your classes and planning for things that will never happen. 
Remember KISS and YAGNI.

You can always come back and insert a _build_widget() method into your class 
when you need it.


> The rationale is that it decouples the class from the queue implementation
> (no evidence or suggestion that we would actually ever want to change impl in
> this case), and makes testing easier.
> 
> I get the underlying principal, and it's one that a strict OOp approach would
> suggest, but my gut feeling is that this is not a pythonic approach at all.

Being able to pass in an optional dependency (the queue, in your example above) 
is a useful, Pythonic thing to do. Making it optional is even more Pythonic. 

But factoring out the call to Queue.Queue() into its own method just for the 
sake of OO purity is not very Pythonic, and *requiring* the caller to call this 
create() method is anti-Pythonic.



-- 
Steven
299792.458 km/s ? not just a good idea, it?s the law!



From arthurhavlicek at gmail.com  Thu Nov  3 02:56:27 2016
From: arthurhavlicek at gmail.com (arthurhavlicek at gmail.com)
Date: Wed, 2 Nov 2016 23:56:27 -0700 (PDT)
Subject: Pre-pep discussion material: in-place equivalents to map and filter
Message-ID: <8a268e42-87c4-4054-bd19-1dc417dd6d01@googlegroups.com>

Hi everybody,

I have an enhancement proposal for Python and, as suggested by PEP 1, am exposing a stub to the mailing list before possibly starting writing a PEP. This is my first message to python mailing list. I hope you will find this content of interest.

Python features a powerful and fast way to create lists through comprehensions. Because of their ease of use and efficiency through native implementation, they are an advantageous alternative to map, filter, and more. However, when used in replacement for an in-place version of these functions, they are sub-optimal, and Python offer no alternative.

This lack of implementation have already been pointed out:
http://stackoverflow.com/questions/4148375/is-there-an-in-place-equivalent-to-map-in-python
Notice the concerns of OPs in his comments to replies in this one:
http://stackoverflow.com/questions/3000461/python-map-in-place
In this answer, developpers here are wondering about performance issues regarding both loop iteration and comprehension:
http://stackoverflow.com/a/1540069/3323394

I would suggest to implement a language-level, in-place version for them. There is severeal motivations for this:

1 - Code readability and reduced redundancy

lst = [ item for item in lst if predicate(item) ]
lst = [ f(item) for item in lst ]

Both these expressions feature redundancy, lst occurs twice and item at least twice. Additionally, the readability is hurt, because one has to dive through the semantics of the comprehension to truely understand I am filtering the list or remapping its values.

Map and filter, although they are more explicit, also feature redundancy. They look OK with functional predicate:

lst = map (f, lst)
lst = filter (predicate, lst)

But are less elegant when using an expression, than one has to convert through a lambda:

lst = map (lambda x: x*5, lst)
lst = filter (lambda x: x%3 == 1, lst)

And perform especially bad in CPython compared to a comprehension.

2 - Efficiency

A language support for these operations to be made in-place could improve the efficiency of this operations through reduced use of memory.


I would propose this syntax. (TODO: find appropriate keywords I guess):

lst.map x: x*5
lst.filter x: x%3 == 1

They can work for dictionaries too.

dct.map k,v: v*5
dct.filter k,v: k+v == 10

The reasonning for the need of a language-level approach is the need for an efficient implementation that would support giving an arbitrary expression and not only a function. Expressions are shorter and, I guess, would be more efficient.


I would gladly appreciate your returns on this, regarding:
1 - Whether a similar proposition has been made
2 - If you find this of any interest at all
3 - If you have a suggestion for improving the proposal

Thanks for reading. Have a nice day


From steve+comp.lang.python at pearwood.info  Thu Nov  3 04:29:44 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Thu, 03 Nov 2016 19:29:44 +1100
Subject: Pre-pep discussion material: in-place equivalents to map and
 filter
References: <8a268e42-87c4-4054-bd19-1dc417dd6d01@googlegroups.com>
Message-ID: <581af57a$0$1585$c3e8da3$5496439d@news.astraweb.com>

On Thursday 03 November 2016 17:56, arthurhavlicek at gmail.com wrote:

[...]
> Python features a powerful and fast way to create lists through
> comprehensions. Because of their ease of use and efficiency through native
> implementation, they are an advantageous alternative to map, filter, and
> more. However, when used in replacement for an in-place version of these
> functions, they are sub-optimal, and Python offer no alternative.


Of course Python offers an alternative: the basic for loop.


for i, x in enumerate(alist):
    alist[i] = func(x)


Not enough of a one-liner for you? Write a helper function:

def inplace_map(func, alist):
    for i, x in enumerate(alist):
        alist[i] = func(x)


In practice, you may even find that except for the most enormous lists (so big 
that memory becomes an issue, so we're talking tens of millions of items) it 
will probably be faster to use a slice and a comprehension:

alist[:] = [func(x) for x in alist]


[...]
> Notice the concerns of OPs in his comments to replies in this one:
> http://stackoverflow.com/questions/3000461/python-map-in-place

What I saw was the OP's *premature optimization*:

"I wanted to use map for performance gains"

"I figured map would be quicker than the list comprehension way"

Although in fairness he does also say:

"I'm working on a big list, and the times we're talking about are seconds in 
difference, which clearly matters"

I'm not necessarily sure that seconds always matters -- if your code takes 90 
seconds, or 96 seconds, who is going to even notice? But let's assume he's 
right and a few seconds difference makes a real difference.

He has three obvious alternatives to waiting until Python 3.7 (the earliest 
such a new feature can be added):

- modify the list in place with a for-loop;
- slice assignment using map;
- slice assignment using list comprehension.


> 1 - Code readability and reduced redundancy
> 
> lst = [ item for item in lst if predicate(item) ]
> lst = [ f(item) for item in lst ]
> 
> Both these expressions feature redundancy, lst occurs twice and item at least
> twice. 

That's not what redundancy means. "Redundancy" doesn't refer to the re-use of 
any arbitrary token. It means doing the same thing twice in two different 
places.


> Additionally, the readability is hurt, because one has to dive through
> the semantics of the comprehension to truely understand I am filtering the
> list or remapping its values.

Nonsense. It is perfectly readable because it is explicit about what is being 
done, unlike some magic method that you have to read the docs to understand 
what it does.

A list comprehension or for-loop is more general and can be combined so you can 
do both:

alist[:] = [func(x) for x in alist if condition(x)]



> Map and filter, although they are more explicit, 

*Less* explicit.

To most people, "map" means the thing that you follow when you are travelling 
in unfamiliar territory, and "filter" means the paper doohickey you put in your 
coffee machine to keep the ground up coffee from ending up in your cup.


> also feature redundancy.
> They look OK with functional predicate:
> 
> lst = map (f, lst)
> lst = filter (predicate, lst)
> 
> But are less elegant when using an expression, than one has to convert
> through a lambda:
> 
> lst = map (lambda x: x*5, lst)
> lst = filter (lambda x: x%3 == 1, lst)

And that's why we have list comprehensions.


> And perform especially bad in CPython compared to a comprehension.

I doubt that.



> 2 - Efficiency
> 
> A language support for these operations to be made in-place could improve the
> efficiency of this operations through reduced use of memory.

*shrug*

Saving memory sometimes costs time.


> I would propose this syntax. (TODO: find appropriate keywords I guess):
> 
> lst.map x: x*5
> lst.filter x: x%3 == 1

I think the chances of Guido accepting new syntax for something as trivial as 
this with three existing solutions is absolutely zero.

I think the chances of Guido accepting new list/dict methods for in place map 
and/or filter is a tiny bit higher than zero.


> The reasonning for the need of a language-level approach is the need for an
> efficient implementation that would support giving an arbitrary expression
> and not only a function. 

We already have three of those: for-loops, list comprehensions, and map.




-- 
Steven
299792.458 km/s ? not just a good idea, it?s the law!



From teppo.pera at gmail.com  Thu Nov  3 04:50:05 2016
From: teppo.pera at gmail.com (teppo.pera at gmail.com)
Date: Thu, 3 Nov 2016 01:50:05 -0700 (PDT)
Subject: constructor classmethods
In-Reply-To: <7f7156ed-e396-460e-8eb1-fb6b17bab069@googlegroups.com>
References: <7f7156ed-e396-460e-8eb1-fb6b17bab069@googlegroups.com>
Message-ID: <02dc88d9-0fd3-416b-8aac-eb2b530d0320@googlegroups.com>

Hello everyone, I'll step into conversation too as I think it is quite important topic. I'd be the one my collegue calls keen to this practice.

Little bit background related to this topic. It all starts from this article:
http://misko.hevery.com/attachments/Guide-Writing%20Testable%20Code.pdf

The guide is written in c++ in mind, yet the concepts stands for any programming language really. Read it through and think about it. If you come back to this topic and say: "yeah, but it's c++", then you haven't understood it. 

Steven is correct in his post, this is Dependency Injection (thanks for seeing through my suggestion, I was starting to think I'm completely crazy, now I'm just partially crazy). Create classmethod is mostly actually for convenience, especially in those cases where the class would require injecting multiple dependencies, which doesn't need to come from outside. I'm fine actually fine using default parameters. I'd like to use __new__ and __init__ here, but I haven't figured out how to do it with those in a way it's comfortable to write and read, which is usually a key point to sell a new practice and also probably a reason why create-classmethod is not popular eiher, regardless it being useful.

My key goal is to maximize the control of object creation, which makes writing tests much much simpler. This is not necessarily OOP, and definitely not strict of such. It comes from the above article and that it is following pretty strictly in a sense. I have at least found it very useful practice to have no logic in constructors. Mock library is definitely powerful mechanism to solve some probelms in Python for testability, but it can (if not carefuly how to write classes you are going to test) make tests very hard to read and if you want to treat your tests as documentation, it is unacceptable.

In order to write a code that just solves the problem, none of those patterns are needed. To solve a problem in a manner that it's possible to return to it later on, then these patterns start to matter.

Br,
Teppo


From hemla21 at gmail.com  Thu Nov  3 04:59:28 2016
From: hemla21 at gmail.com (Heli)
Date: Thu, 3 Nov 2016 01:59:28 -0700 (PDT)
Subject: Reading Fortran Ascii output using python
In-Reply-To: <mailman.28.1477940606.17234.python-list@python.org>
References: <bdaa6028-e7a8-4b23-ab4e-a3f2d9bc2089@googlegroups.com>
 <58177f8d$0$946$e4fe514c@news.xs4all.nl>
 <a26e5472-5630-4898-80d8-aa68fbc6d36f@googlegroups.com>
 <1ede9bc5-5897-b1cc-3875-46bc13c182a9@mrabarnett.plus.com>
 <mailman.28.1477940606.17234.python-list@python.org>
Message-ID: <6057ba03-58eb-4274-8c90-e2a90c5b8cad@googlegroups.com>

On Monday, October 31, 2016 at 8:03:53 PM UTC+1, MRAB wrote:
> On 2016-10-31 17:46, Heli wrote:
> > On Monday, October 31, 2016 at 6:30:12 PM UTC+1, Irmen de Jong wrote:
> >> On 31-10-2016 18:20, Heli wrote:
> >> > Hi all,
> >> >
> >> > I am trying to read an ascii file written in Fortran90 using python. I am reading this file by opening the input file and then reading using:
> >> >
> >> > inputfile.readline()
> >> >
> >> > On each line of the ascii file I have a few numbers like this:
> >> >
> >> > line 1: 1
> >> > line 2: 1000.834739 2000.38473 3000.349798
> >> > line 3: 1000 2000 5000.69394 99934.374638 54646.9784
> >> >
> >> > The problem is when I have more than 3 numbers on the same line such as line 3, python seems to read this using two reads. This makes the above example will be read like this:
> >> >
> >> > line 1: 1
> >> > line 2: 1000.834739 2000.38473 3000.349798
> >> > line 3: 1000 2000 5000.69394
> >> > line 4: 99934.374638 54646.9784
> >> >
> >> > How can I fix this for each fortran line to be read correctly using python?
> >> >
> >> > Thanks in Advance for your help,
> >> >
> >> >
> >>
> >> You don't show any code so it's hard to say what is going on.
> >> My guess is that your file contains spurious newlines and/or CRLF combinations.
> >>
> >> Try opening the file in universal newline mode and see what happens?
> >>
> >> with open("fortranfile.txt", "rU") as f:
> >>     for line in f:
> >>         print("LINE:", line)
> >>
> >>
> >> Irmen
> >
> > Thanks Irmen,
> >
> > I tried with "rU" but that did not make a difference. The problem is a line that with one single write statement in my fortran code :
> >
> > write(UNIT=9,FMT="(99g20.8)")  value
> >
> > seems to be read in two python inputfile.readline().
> >
> > Any ideas how I should be fixing this?
> >
> > Thanks,
> >
> What is actually in the file?
> 
> Try opening it in binary mode and print using the ascii function:
> 
>      with open("fortranfile.txt", "rb") as f:
>          contents = f.read()
> 
>      print("CONTENTS:", ascii(contents))

Thanks guys, 
I solved the problem on the Fortran side. Some lines contained new lines characters that I fixed by setting format descriptors in Fortran.

Thanks for your help, 


From fabiofz at gmail.com  Thu Nov  3 05:44:39 2016
From: fabiofz at gmail.com (Fabio Zadrozny)
Date: Thu, 3 Nov 2016 07:44:39 -0200
Subject: PyDev 5.3.1 Released
Message-ID: <CANXBEFoe8CR+fd5skCqe04Sn7BG1jEG7-U80GVqH8P+v7jAztw@mail.gmail.com>

Release Highlights:
-------------------------------

* **Important** PyDev now requires Java 8 and Eclipse 4.6 (Neon) onwards.

    * PyDev 5.2.0 is the last release supporting Eclipse 4.5 (Mars).

* **Code Completion**

    * Substring completions are **on by default** (may be turned off in the
code-completion preferences).
    * Fixed issue with code-completion using from..import..as aliases.

* **Others**

    * Auto-fix imports with Ctrl+Shift+O properly sorts items based on the
same sorting improvements for code-completion.
    * When fixing unresolved import (with Ctrl+1) it properly resolves
dependent projects (bugfix for regression in 5.3.0).
    * **async** and **await** keywords are properly highlighted.
    * **async** blocks properly auto-indented.
    * In PEP 448 list unpack variable was not being marked as a "Load"
variable (which made the code analysis yield false positives).

What is PyDev?
---------------------------

PyDev is an open-source Python IDE on top of Eclipse for Python, Jython and
IronPython development.

It comes with goodies such as code completion, syntax highlighting, syntax
analysis, code analysis, refactor, debug, interactive console, etc.

Details on PyDev: http://pydev.org
Details on its development: http://pydev.blogspot.com


What is LiClipse?
---------------------------

LiClipse is a PyDev standalone with goodies such as support for Multiple
cursors, theming, TextMate bundles and a number of other languages such as
Django Templates, Jinja2, Kivy Language, Mako Templates, Html, Javascript,
etc.

It's also a commercial counterpart which helps supporting the development
of PyDev.

Details on LiClipse: http://www.liclipse.com/



Cheers,

--
Fabio Zadrozny
------------------------------------------------------
Software Developer

LiClipse
http://www.liclipse.com

PyDev - Python Development Environment for Eclipse
http://pydev.org
http://pydev.blogspot.com

PyVmMonitor - Python Profiler
http://www.pyvmmonitor.com/


From hemla21 at gmail.com  Thu Nov  3 06:08:23 2016
From: hemla21 at gmail.com (Heli)
Date: Thu, 3 Nov 2016 03:08:23 -0700 (PDT)
Subject: data interpolation
Message-ID: <2db289bc-274a-4e05-8d4a-07822ea97b24@googlegroups.com>

Hi, 

I have a question about data interpolation using python. I have a big ascii file containg data in the following format and around 200M points. 

id, xcoordinate, ycoordinate, zcoordinate

then I have a second file containing data in the following format, ( 2M values) 

id, xcoordinate, ycoordinate, zcoordinate, value1, value2, value3,..., valueN

I would need to get values for x,y,z coordinates of file 1 from values of file2.  

I don?t know whether my data in file1 and 2 is from structured or unstructured grid source. I was wondering which interpolation module either from scipy or scikit-learn you recommend me to use?

I would also appreciate if you could recommend me some sample example/reference. 

Thanks in Advance for your help, 


From ned at nedbatchelder.com  Thu Nov  3 06:42:02 2016
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Thu, 3 Nov 2016 03:42:02 -0700 (PDT)
Subject: Pre-pep discussion material: in-place equivalents to map and
 filter
In-Reply-To: <581af57a$0$1585$c3e8da3$5496439d@news.astraweb.com>
References: <8a268e42-87c4-4054-bd19-1dc417dd6d01@googlegroups.com>
 <581af57a$0$1585$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <b136a64f-066e-454f-9508-5a04b704ab00@googlegroups.com>

On Thursday, November 3, 2016 at 4:30:00 AM UTC-4, Steven D'Aprano wrote:
> On Thursday 03 November 2016 17:56, arthurhavlicek at gmail.com wrote:
> > I would propose this syntax. (TODO: find appropriate keywords I guess):
> > 
> > lst.map x: x*5
> > lst.filter x: x%3 == 1
> 
> I think the chances of Guido accepting new syntax for something as trivial as 
> this with three existing solutions is absolutely zero.
> 
> I think the chances of Guido accepting new list/dict methods for in place map 
> and/or filter is a tiny bit higher than zero.


To my eyes, this proposed syntax is completely foreign. "lst.map" looks
like attribute or method access on lst, but there's no operation on the
result, or function call.  They implicitly assign back to lst, with no
recognizable syntax to indicate that they do (= or "as" is the usual
marker).

While mapping and filtering are common operations, I'm not sure mapping
and filtering and then reassigning back to the original sequence is
especially common.  It's certainly not common enough to deserve completely
new syntax when the existing methods already exist.

Part of the problem here is that you value explicitness, but also are
trying to reduce redundancy.  When you say that lst occurs twice in
your examples, what I see is that it occurs twice because it's being
used for two different things: it is the source of the data, and it is
also the target for the result. I think it is good to have it appear
twice in this case, especially since there's no reason to think it will
usually be used for both purposes.  How do I do exactly the same data
manipulation, but then assign it to lst2 because I belatedly realized
I wanted both the before and after list?  Under your proposed syntax,
I would have to completely rewrite the line to use a different filtering
mechanism because now I need to unbundle the filtering and the assignment.
That seems unfortunate.

You've done the right thing by bringing the proposal here.  I think it
is useful to see how people approach the language, and where they want
changes.  Discussing the pros and cons is a good way to get a deeper
appreciation both for the language and the rationale for its design.
But I don't think this proposal has a chance of moving forward.

--Ned.


From kenansharon at gmail.com  Thu Nov  3 09:10:49 2016
From: kenansharon at gmail.com (kenansharon at gmail.com)
Date: Thu, 3 Nov 2016 06:10:49 -0700 (PDT)
Subject: Problems with read_eager and Telnet
In-Reply-To: <f4532ead-730e-45b6-accf-7a3722ae72e7@u14g2000vbg.googlegroups.com>
References: <f4532ead-730e-45b6-accf-7a3722ae72e7@u14g2000vbg.googlegroups.com>
Message-ID: <1fe47b58-9498-4ee5-9fa1-225a3c31e212@googlegroups.com>

On Monday, 28 February 2011 10:54:56 UTC-5, Robi  wrote:
> Hi everybody,
>  I'm totally new to Python but well motivated :-)
> 
> I'm fooling around with Python in order to interface with FlightGear
> using a telnet connection.
> 
> I can do what I had in mind (send some commands and read output from
> Flightgear using the telnetlib) with a read_until() object to catch
> every output line I need, but it proved to be very slow (it takes 1/10
> of a sec for every read_until().
> 
> I tried using the read_eager() object and it's waaaayyyy faster (it
> does the job in 1/100 of a sec, maybe more, I didn't tested) but it
> gives me problems, it gets back strange strings, repeated ones,
> partially broken ones, well ... I don't know what's going on with it.
> 
> You see, I don't know telnet (the protocol) very good, I'm very new to
> Python and Python's docs are not very specific about that read_eager(9
> stuff.
> 
> Could someone point me to some more documentation about that? or at
> least help me in getting a correct idea of what's going on with
> read_eager()?
> 
> I'm going on investigating but a help from here would be very
> appreciated :-)
> 
> Thanks in advance,
>    Roberto

Say a can someone explain for me how read_eager and read _very_eager decide when to stop reading date from a socket?

If the command sent causes the server to reply with multiple lines how do the Python functions decide when to stop accepting new data from the socket?


From rosuav at gmail.com  Thu Nov  3 10:05:07 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 4 Nov 2016 01:05:07 +1100
Subject: Pre-pep discussion material: in-place equivalents to map and
 filter
In-Reply-To: <581af57a$0$1585$c3e8da3$5496439d@news.astraweb.com>
References: <8a268e42-87c4-4054-bd19-1dc417dd6d01@googlegroups.com>
 <581af57a$0$1585$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPTjJmrQdo2J2ztRz-waim2rsaugi-CJDzOrFe441-OqgoS9Pg@mail.gmail.com>

On Thu, Nov 3, 2016 at 7:29 PM, Steven D'Aprano
<steve+comp.lang.python at pearwood.info> wrote:
>> lst = map (lambda x: x*5, lst)
>> lst = filter (lambda x: x%3 == 1, lst)
>> And perform especially bad in CPython compared to a comprehension.
>
> I doubt that.
>

It's entirely possible. A list comp involves one function call (zero
in Py2), but map/lambda involves a function call per element in the
list. Function calls have overhead.

Arthur, I would suggest looking at what numpy and pandas do. When
you're working with ridiculously large data sets, they absolutely
shine; and if you're not working with that much data, the performance
of map or a list comp is unlikely to be significant. If the numpy
folks have a problem that can't be solved without new syntax, then a
proposal can be brought to the core (like matmul, which was approved
and implemented).

ChrisA


From ethan at stoneleaf.us  Thu Nov  3 10:45:28 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 03 Nov 2016 07:45:28 -0700
Subject: constructor classmethods
In-Reply-To: <02dc88d9-0fd3-416b-8aac-eb2b530d0320@googlegroups.com>
References: <7f7156ed-e396-460e-8eb1-fb6b17bab069@googlegroups.com>
 <02dc88d9-0fd3-416b-8aac-eb2b530d0320@googlegroups.com>
Message-ID: <581B4D88.9010803@stoneleaf.us>

On 11/03/2016 01:50 AM, teppo.pera at gmail.com wrote:

> The guide is written in c++ in mind, yet the concepts stands for any
>  programming language really. Read it through and think about it. If
>  you come back to this topic and say: "yeah, but it's c++", then you
>  haven't understood it.

The ideas (loose coupling, easy testing) are certainly applicable in Python -- the specific methods talked about in that paper, however, are not.

To go back to the original example:

def __init__(self, ...):
     self.queue = Queue()

we have several different (easy!) ways to do dependency injection:

* inject a mock Queue into the module
* make queue a default parameter

If it's just testing, go with the first option:

import the_module_to_test
the_module_to_test.Queue = MockQueue

and away you go.

If the class in question has legitimate, non-testing, reasons to specify different Queues, then make it a default argument instead:

def __init__(self, ..., queue=None):
     if queue is None:
         queue = Queue()
     self.queue = queue

or, if it's just for testing but you don't want to hassle injecting a MockQueue into the module itself:

def __init__(self, ..., _queue=None):
     if _queue is None:
         _queue = Queue()
     self.queue = _queue

or, if the queue is only initialized (and not used) during __init__ (so you can replace it after construction with no worries):

class Example:
     def __init__(self, ...):
         self.queue = Queue()

ex = Example()
ex.queue = MockQueue()
# proceed with test

The thing each of those possibilities have in common is that the normal use-case of just creating the thing and moving on is the very simple:

my_obj = Example(...)

To sum up:  your concerns are valid, but using c++ (and many other language) idioms in Python does not make good Python code.

--
~Ethan~


From rosuav at gmail.com  Thu Nov  3 10:47:03 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 4 Nov 2016 01:47:03 +1100
Subject: constructor classmethods
In-Reply-To: <02dc88d9-0fd3-416b-8aac-eb2b530d0320@googlegroups.com>
References: <7f7156ed-e396-460e-8eb1-fb6b17bab069@googlegroups.com>
 <02dc88d9-0fd3-416b-8aac-eb2b530d0320@googlegroups.com>
Message-ID: <CAPTjJmr3c9A3ifpp-Y-5js2v7gY+Bo7us5YX89L-bTu-8sMVUw@mail.gmail.com>

On Thu, Nov 3, 2016 at 7:50 PM,  <teppo.pera at gmail.com> wrote:
> Little bit background related to this topic. It all starts from this article:
> http://misko.hevery.com/attachments/Guide-Writing%20Testable%20Code.pdf
>
> The guide is written in c++ in mind, yet the concepts stands for any programming language really. Read it through and think about it. If you come back to this topic and say: "yeah, but it's c++", then you haven't understood it.

I don't have a problem with something written for C++ (though I do
have a problem with a thirty-eight page document on how to make your
code testable - TLDR), but do bear in mind that a *lot* of C++ code
can be simplified when it's brought to Python. One Python feature that
C++ doesn't have, mentioned already in this thread, is the way you can
have a ton of parameters with defaults, and you then specify only
those you want, as keyword args:

def __init__(self, important_arg1, important_arg2,
        queue=None, cache_size=50, whatever=...):
    pass

MyClass("foo", 123, cache_size=75)

I can ignore all the arguments that don't matter, and provide only the
one or two that I actually need to change. Cognitive load is
drastically reduced, compared to the "alternative constructor"
pattern, where I have to remember not to construct anything in the
normal way.

You can't do that in C++, so it's not going to be mentioned in that
document, but it's an excellent pattern to follow.

ChrisA


From tjreedy at udel.edu  Thu Nov  3 11:04:36 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 3 Nov 2016 11:04:36 -0400
Subject: Pre-pep discussion material: in-place equivalents to map and
 filter
In-Reply-To: <581af57a$0$1585$c3e8da3$5496439d@news.astraweb.com>
References: <8a268e42-87c4-4054-bd19-1dc417dd6d01@googlegroups.com>
 <581af57a$0$1585$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <nvfjm3$ctg$1@blaine.gmane.org>

On 11/3/2016 4:29 AM, Steven D'Aprano wrote:

> Nonsense. It is perfectly readable because it is explicit about what is being
> done, unlike some magic method that you have to read the docs to understand
> what it does.

Agreed.

> A list comprehension or for-loop is more general and can be combined so you can
> do both:
>
> alist[:] = [func(x) for x in alist if condition(x)]

The qualifier 'list' is not needed.  The right hand side of slice 
assignment just has to be an iterable.  So a second interation to build 
an intermediate list is not required.

alist[:] = (func(x) for x in alist if condition(x))

The parentheses around the generator expression are required here. 
(Steven, I know you know that, but not everyone else will.)

-- 
Terry Jan Reedy



From tolbarusorin at gmail.com  Thu Nov  3 11:26:09 2016
From: tolbarusorin at gmail.com (Constantin Sorin)
Date: Thu, 3 Nov 2016 08:26:09 -0700 (PDT)
Subject: Python Dice Game/Need help with my script/looping!
Message-ID: <e69bcfb9-0ee3-4570-843a-130caa16b4aa@googlegroups.com>

Hello,I recently started to make a dice game in python.Everything was nice and beautiful,until now.My problem is that when I try to play and I win or lost or it's equal next time it will continue only with that.
Exemple:
Enter name >> Sorin
Money = 2
Bet >> 2
You won!
Money 4
Bet >> 2
You won!
and it loops like this :/

Here is the code:

import time
import os
import random
os = os.system
os("clear")

print "What is your name?"
name = raw_input(">>")

def lost():
	print "Yoy lost the game!Wanna Play again?Y/N"
	ch = raw_input(">>")
	if ch == "Y":
		game()
	elif ch == "N":
		exit()



def game():
	os("clear")
	a = random.randint(1,6)
	b = random.randint(1,6)
	c = random.randint(1,6)
	d = random.randint(1,6)
	e = a + b
	f = c + d
	money = 2
	while money > 0:
		print "Welcome to FireDice %s!" %name
		print "Your money: %s$" %money
		print "How much do you bet?"
		bet = input(">>")
		if e > f:
			print "you won!"
			money = money + bet
		elif e < f:
			print "you lost"
			money = money - bet
		else:
			print "?"

		print money
	lost()	

game()


From tolbarusorin at gmail.com  Thu Nov  3 11:27:58 2016
From: tolbarusorin at gmail.com (Constantin Sorin)
Date: Thu, 3 Nov 2016 08:27:58 -0700 (PDT)
Subject: Python Dice Game/Need help with my script/looping!
In-Reply-To: <e69bcfb9-0ee3-4570-843a-130caa16b4aa@googlegroups.com>
References: <e69bcfb9-0ee3-4570-843a-130caa16b4aa@googlegroups.com>
Message-ID: <ebd5e2c5-6c5b-4ff6-870d-5f56af79db90@googlegroups.com>

I use Linux and python 2.7.12


From fillmore_remove at hotmail.com  Thu Nov  3 12:18:06 2016
From: fillmore_remove at hotmail.com (Fillmore)
Date: Thu, 3 Nov 2016 12:18:06 -0400
Subject: need some kind of "coherence index" for a group of strings
Message-ID: <nvfnvu$1bc1$1@gioia.aioe.org>


Hi there, apologies for the generic question. Here is my problem let's 
say that I have a list of lists of strings.

list1:    #strings are sort of similar to one another

   my_nice_string_blabla
   my_nice_string_blqbli
   my_nice_string_bl0bla
   my_nice_string_aru


list2:    #strings are mostly different from one another

   my_nice_string_blabla
   some_other_string
   yet_another_unrelated string
   wow_totally_different_from_others_too


I would like an algorithm that can look at the strings and determine 
that strings in list1 are sort of similar to one another, while the 
strings in list2 are all different.
Ideally, it would be nice to have some kind of 'coherence index' that I 
can exploit to separate lists given a certain threshold.

I was about to concoct something using levensthein distance, but then I 
figured that it would be expensive to compute and I may be reinventing 
the wheel.

Thanks in advance to python masters that may have suggestions...





From walters.justin01 at gmail.com  Thu Nov  3 12:34:14 2016
From: walters.justin01 at gmail.com (justin walters)
Date: Thu, 3 Nov 2016 09:34:14 -0700
Subject: need some kind of "coherence index" for a group of strings
In-Reply-To: <nvfnvu$1bc1$1@gioia.aioe.org>
References: <nvfnvu$1bc1$1@gioia.aioe.org>
Message-ID: <CAO1D73F_6xLurW=r-rDTRiDcFPjOKaj04wc9h-F=5TWV0dwUpg@mail.gmail.com>

On Thu, Nov 3, 2016 at 9:18 AM, Fillmore <fillmore_remove at hotmail.com>
wrote:

>
> Hi there, apologies for the generic question. Here is my problem let's say
> that I have a list of lists of strings.
>
> list1:    #strings are sort of similar to one another
>
>   my_nice_string_blabla
>   my_nice_string_blqbli
>   my_nice_string_bl0bla
>   my_nice_string_aru
>
>
> list2:    #strings are mostly different from one another
>
>   my_nice_string_blabla
>   some_other_string
>   yet_another_unrelated string
>   wow_totally_different_from_others_too
>
>
> I would like an algorithm that can look at the strings and determine that
> strings in list1 are sort of similar to one another, while the strings in
> list2 are all different.
> Ideally, it would be nice to have some kind of 'coherence index' that I
> can exploit to separate lists given a certain threshold.
>
> I was about to concoct something using levensthein distance, but then I
> figured that it would be expensive to compute and I may be reinventing the
> wheel.
>
> Thanks in advance to python masters that may have suggestions...
>
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>


When you say similar, do you mean similar in the amount of duplicate
words/letters? Or were you more interested
in similar sentence structure?


From steve+python at pearwood.info  Thu Nov  3 13:00:39 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Fri, 04 Nov 2016 04:00:39 +1100
Subject: Pre-pep discussion material: in-place equivalents to map and
 filter
References: <8a268e42-87c4-4054-bd19-1dc417dd6d01@googlegroups.com>
 <581af57a$0$1585$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmrQdo2J2ztRz-waim2rsaugi-CJDzOrFe441-OqgoS9Pg@mail.gmail.com>
 <mailman.17.1478181910.7975.python-list@python.org>
Message-ID: <581b6d3a$0$1583$c3e8da3$5496439d@news.astraweb.com>

On Fri, 4 Nov 2016 01:05 am, Chris Angelico wrote:

> On Thu, Nov 3, 2016 at 7:29 PM, Steven D'Aprano
> <steve+comp.lang.python at pearwood.info> wrote:
>>> lst = map (lambda x: x*5, lst)
>>> lst = filter (lambda x: x%3 == 1, lst)
>>> And perform especially bad in CPython compared to a comprehension.
>>
>> I doubt that.
>>
> 
> It's entirely possible. A list comp involves one function call (zero
> in Py2), but map/lambda involves a function call per element in the
> list. Function calls have overhead.

I don't know why you think that list comps don't involve function calls.

Here's some timing results using 3.5 on my computer. For simplicity, so
folks can replicate the test themselves, here's the timing code:


from timeit import Timer
setup = """data = list(range(10000))
def func(x):  # simulate some calculation
    return {x+1: x**2}
"""
t1 = Timer('[func(a) for a in data]', setup=setup)
t2 = Timer('list(map(func, data))', setup=setup)



And here's the timing results on my machine:

py> min(t1.repeat(number=1000, repeat=7))  # list comp
18.2571472954005
py> min(t2.repeat(number=1000, repeat=7))  # map
18.157311914488673

So there's hardly any difference, but map() is about 0.5% faster in this
case.

Now, it is true that *if* you can write the list comprehension as a direct
calculation in an expression instead of a function call:

    [a+1 for a in data]

*and* compare it to map using a function call:

    map(lambda a: a+1, data)


then the overhead of the function call in map() may be significant. But the
extra cost is effectively just a multiplier. It isn't going to change
the "Big Oh" behaviour. Here's an example:

t3 = Timer('[a+1 for a in data]', setup=setup)
t4 = Timer('list(map(lambda a: a+1, data))', setup=setup)
extra = """from functools import partial
from operator import add
"""
t5 = Timer('list(map(partial(add, 1), data))', setup=setup+extra)

And the timing results:

py> min(t3.repeat(number=1000, repeat=7))  # list comp with expression
2.6977453008294106
py> min(t4.repeat(number=1000, repeat=7))  # map with function
4.280411267653108
py> min(t5.repeat(number=1000, repeat=7))  # map with partial
3.446241058409214

So map() here is less than a factor of two slower. I wouldn't call
that "especially bad" -- often times, a factor of two is not important.
What really hurts is O(N**2) performance, or worse.

So at worst, map() is maybe half as fast as a list comprehension, and at
best, its perhaps a smidgen faster. I would expect around the same
performance, differing only by a small multiplicative factor: I wouldn't
expect one to be thousands or even tens of times slower that the other.



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From jladasky at itu.edu  Thu Nov  3 13:49:00 2016
From: jladasky at itu.edu (jladasky at itu.edu)
Date: Thu, 3 Nov 2016 10:49:00 -0700 (PDT)
Subject: need some kind of "coherence index" for a group of strings
In-Reply-To: <nvfnvu$1bc1$1@gioia.aioe.org>
References: <nvfnvu$1bc1$1@gioia.aioe.org>
Message-ID: <b926915c-7862-4fe3-a74b-ae11048e2390@googlegroups.com>

The Levenshtein distance is a very precise definition of dissimilarity between sequences.  It specifies the minimum number of single-element edits you would need to change one sequence into another.  You are right that it is fairly expensive to compute.

But you asked for an algorithm that would determine whether groups of strings are "sort of similar".  How imprecise can you be?  An analysis of the frequency of each individual character in a string might be good enough for you.


From ethan at stoneleaf.us  Thu Nov  3 15:02:58 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 03 Nov 2016 12:02:58 -0700
Subject: constructor classmethods
In-Reply-To: <581B4D88.9010803@stoneleaf.us>
References: <7f7156ed-e396-460e-8eb1-fb6b17bab069@googlegroups.com>
 <02dc88d9-0fd3-416b-8aac-eb2b530d0320@googlegroups.com>
 <581B4D88.9010803@stoneleaf.us>
Message-ID: <581B89E2.6020209@stoneleaf.us>

On 11/03/2016 07:45 AM, Ethan Furman wrote:
> On 11/03/2016 01:50 AM, teppo.pera at gmail.com wrote:
>
>> The guide is written in c++ in mind, yet the concepts stands for any
>>  programming language really. Read it through and think about it. If
>>  you come back to this topic and say: "yeah, but it's c++", then you
>>  haven't understood it.
>
> The ideas (loose coupling, easy testing) are certainly applicable in Python -- the specific methods talked about in that paper, however, are not.

Speaking specifically about the gyrations needed for the sole purpose of testing.

The paper had a lot of good things to say about decoupling, and in that light if the class in question should work with any Queue, then it should be passed in -- however, if it's an implementation detail, then it shouldn't.

--
~Ethan~


From neilc at norwich.edu  Thu Nov  3 16:08:34 2016
From: neilc at norwich.edu (Neil D. Cerutti)
Date: Thu, 03 Nov 2016 16:08:34 -0400
Subject: need some kind of "coherence index" for a group of strings
In-Reply-To: <b926915c-7862-4fe3-a74b-ae11048e2390@googlegroups.com>
References: <nvfnvu$1bc1$1@gioia.aioe.org>
 <b926915c-7862-4fe3-a74b-ae11048e2390@googlegroups.com>
Message-ID: <nvg5g0$t7n$1@blaine.gmane.org>

On 11/3/2016 1:49 PM, jladasky at itu.edu wrote:
> The Levenshtein distance is a very precise definition of dissimilarity between sequences.  It specifies the minimum number of single-element edits you would need to change one sequence into another.  You are right that it is fairly expensive to compute.
>
> But you asked for an algorithm that would determine whether groups of strings are "sort of similar".  How imprecise can you be?  An analysis of the frequency of each individual character in a string might be good enough for you.

I also once used a Levenshtein distance algo in Python (code snippet 
D0DE4716-B6E6-4161-9219-2903BF8F547F) to compare names of students (it 
worked, but turned out to not be what I needed), but you may also be 
able to use some items "off the shelf" from Python's difflib.

-- 
Neil Cerutti



From arthurhavlicek at gmail.com  Thu Nov  3 16:29:06 2016
From: arthurhavlicek at gmail.com (Arthur Havlicek)
Date: Thu, 3 Nov 2016 21:29:06 +0100
Subject: Pre-pep discussion material: in-place equivalents to map and
 filter
In-Reply-To: <581b6d3a$0$1583$c3e8da3$5496439d@news.astraweb.com>
References: <8a268e42-87c4-4054-bd19-1dc417dd6d01@googlegroups.com>
 <581af57a$0$1585$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmrQdo2J2ztRz-waim2rsaugi-CJDzOrFe441-OqgoS9Pg@mail.gmail.com>
 <mailman.17.1478181910.7975.python-list@python.org>
 <581b6d3a$0$1583$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPpswAJF7PzVb1nyqD0fj=2SuW7pT6QNrH_vz1+CVSNGU-Pv2Q@mail.gmail.com>

I understand that, the cost of change is such that it's very unlikely
something like this ever goes into Python, but I feel like the interest of
the proposition is being underestimated here, that's why I'm going to argue
a few points and give a bit more context as needed.

> While mapping and filtering are common operations, I'm not sure mapping
> and filtering and then reassigning back to the original sequence is
> especially common.

It depends of your context. On the last 3 months, I stumbled across this at
least 3 times, which is 3 times more than I used a lambda or a metaclass or
a decorator or other fancy language feature that we simply avoid whenever
possible. It also happened to my collegue. I remember these examples
because we had a bit of humour about how nice can be inlined ternaries
inside comprehensions, but I could be missing a lot more.

The reason I'm being especially impacted by this is because I am maintainer
of a decent-sized Python application (~50-100K lines of code) that
extensively uses lists and dictionaries. We value "low level" data
manipulation and efficiency a lot more than complex, non-obvious
constructs. In other contexts, it may be very different. Note that my
context is only relevant for illustration here, I don't expect a feature to
save me since we are currently shipping to Centos 6 and thus will not see
the light of Python 3.7 in the next 10 years (optimistic estimation).

> Arthur, I would suggest looking at what numpy and pandas do.

In my example context, their benefits can't apply, because I'm not going to
rewrite code that uses lists for them to uses np.array instead for example.
Although the performance boost is likely to be bigger if used properly, I
would have prefered a lesser boost that comes for (almost) free.

Like most Python programmers, I'm not in the case of needing a performance
boost very bad, but that does not mean I disregard performance entirely.
The need of performance is not so binary that it either don't matter at all
or is enough to motivate a rewrite.

> To my eyes, this proposed syntax is completely foreign

I must admit I don't have much imagination for syntax proposals...all that
mattered to me here was to make it clear you are doing an in-place
modification. Feel free to completely ignore that part. Any proposal
welcomed of course.
About Readability & Redundancy

I have misused the terms here, but I wasn't expecting so much nitpicking. I
should have used the term maintenability, because that one is bland and
subjective enough that nobody would have noticed :D

How about "I find that cooler." Good enough ?

In a less sarcastic tone:

What I truely meant here is that when you contain the behavior of your code
inside a specific keyword or syntax, you are making your intentions clear
to the reader. It may be harder for him to gain access to the knowledge in
the first place, but makes it easier over time.

Famous example:

When you learned programming, you may have had no idea what "+=" was doing,
but now you do, you probably rate "a += 2" syntax to be much better than "a
= a + 2". You make the economy of a token, but more important, you make
your intentions clearer because "+=" rings a bell, wihle "=" is a more
generic syntax with a broader meaning.

> So map() here is less than a factor of two slower. I wouldn't call
> that "especially bad" -- often times, a factor of two is not important.
> What really hurts is O(N**2) performance, or worse.

When you evaluate your application bottleneck or your average daily
algorithmic evaluation, perhaps. When regarding language core features we
are not on the same scale. If a language X is 2 times faster than a
language Y to do the same task, that's a huge seller, and is of real
importance.


From bgailer at gmail.com  Thu Nov  3 17:31:49 2016
From: bgailer at gmail.com (Bob Gailer)
Date: Thu, 3 Nov 2016 17:31:49 -0400
Subject: Python Dice Game/Need help with my script/looping!
In-Reply-To: <e69bcfb9-0ee3-4570-843a-130caa16b4aa@googlegroups.com>
References: <e69bcfb9-0ee3-4570-843a-130caa16b4aa@googlegroups.com>
Message-ID: <CAP1rxO7GnZshwnBNHfrduoBL_s334npiHL3Pim39yL=qr2hi5g@mail.gmail.com>

On Nov 3, 2016 11:30 AM, "Constantin Sorin" <tolbarusorin at gmail.com> wrote:
>
> Hello,I recently started to make a dice game in python.Everything was
nice and beautiful,until now.My problem is that when I try to play and I
win or lost or it's equal next time it will continue only with that.
> Exemple:
> Enter name >> Sorin
> Money = 2
> Bet >> 2
> You won!
> Money 4
> Bet >> 2
> You won!
> and it loops like this :/
>

What are the rules of the game?

> Here is the code:
>
> import time
> import os
> import random
> os = os.system
> os("clear")
>
> print "What is your name?"
> name = raw_input(">>")
>
> def lost():
>         print "Yoy lost the game!Wanna Play again?Y/N"
>         ch = raw_input(">>")
>         if ch == "Y":
>                 game()

When you call a function from within that function you are using recursion.
This is not what recursion is intended for. If you play long enough you
will run out of memory.

The proper way to handle this is to put the entire body of game() in a
while loop.

>         elif ch == "N":
>                 exit()
>
>
>
> def game():
>         os("clear")
>         a = random.randint(1,6)
>         b = random.randint(1,6)
>         c = random.randint(1,6)
>         d = random.randint(1,6)
>         e = a + b
>         f = c + d
>         money = 2
>         while money > 0:
>                 print "Welcome to FireDice %s!" %name
>                 print "Your money: %s$" %money
>                 print "How much do you bet?"
>                 bet = input(">>")
>                 if e > f:
>                         print "you won!"
>                         money = money + bet
>                 elif e < f:
>                         print "you lost"
>                         money = money - bet
>                 else:
>                         print "?"
>
>                 print money
>         lost()

Since the values of e and f are not changed in the loop he will continue to
get the same thing.
>
> game()
> --
> https://mail.python.org/mailman/listinfo/python-list


From rosuav at gmail.com  Thu Nov  3 17:34:34 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 4 Nov 2016 08:34:34 +1100
Subject: Pre-pep discussion material: in-place equivalents to map and
 filter
In-Reply-To: <581b6d3a$0$1583$c3e8da3$5496439d@news.astraweb.com>
References: <8a268e42-87c4-4054-bd19-1dc417dd6d01@googlegroups.com>
 <581af57a$0$1585$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmrQdo2J2ztRz-waim2rsaugi-CJDzOrFe441-OqgoS9Pg@mail.gmail.com>
 <mailman.17.1478181910.7975.python-list@python.org>
 <581b6d3a$0$1583$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPTjJmp9Sr1=-FoOz=aqVxXaFQ3AjZWUH4jq868nJEFdhbH4yA@mail.gmail.com>

On Fri, Nov 4, 2016 at 4:00 AM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> On Fri, 4 Nov 2016 01:05 am, Chris Angelico wrote:
>
>> On Thu, Nov 3, 2016 at 7:29 PM, Steven D'Aprano
>> <steve+comp.lang.python at pearwood.info> wrote:
>>>> lst = map (lambda x: x*5, lst)
>>>> lst = filter (lambda x: x%3 == 1, lst)
>>>> And perform especially bad in CPython compared to a comprehension.
>>>
>>> I doubt that.
>>>
>>
>> It's entirely possible. A list comp involves one function call (zero
>> in Py2), but map/lambda involves a function call per element in the
>> list. Function calls have overhead.
>
> I don't know why you think that list comps don't involve function calls.

List comps themselves involve one function call (zero in Py2). What
you do inside the expression is your business. Do you agree that list
comps don't have the overhead of opening and closing files?

files = "/etc/hostname", "/etc/resolv.conf", ".bashrc"
contents = [open(fn).read() for fn in files]

In a comparison between comprehensions and map, this cost is
irrelevant, unless your point is that "they're all pretty quick".

> Here's some timing results using 3.5 on my computer. For simplicity, so
> folks can replicate the test themselves, here's the timing code:
>
>
> from timeit import Timer
> setup = """data = list(range(10000))
> def func(x):  # simulate some calculation
>     return {x+1: x**2}
> """
> t1 = Timer('[func(a) for a in data]', setup=setup)
> t2 = Timer('list(map(func, data))', setup=setup)

This is very different from the original example, about which the OP
said that map performs badly, and you doubted it. In that example, the
list comp includes the expression directly, whereas map (by its
nature) must use a function. The "inline expression" of a
comprehension is more efficient than the "inline expression" of a
lambda function given to map.

> And here's the timing results on my machine:
>
> py> min(t1.repeat(number=1000, repeat=7))  # list comp
> 18.2571472954005
> py> min(t2.repeat(number=1000, repeat=7))  # map
> 18.157311914488673
>
> So there's hardly any difference, but map() is about 0.5% faster in this
> case.

Right. As has often been stated, map is perfectly efficient, *if* the
body of the comprehension would be simply a function call, nothing
more. You can map(str, stuff) to stringify a bunch of things. Nice.
But narrow, and not the code that was being discussed.

> Now, it is true that *if* you can write the list comprehension as a direct
> calculation in an expression instead of a function call:
>
>     [a+1 for a in data]
>
> *and* compare it to map using a function call:
>
>     map(lambda a: a+1, data)
>
>
> then the overhead of the function call in map() may be significant.

Thing is, this is extremely common. How often do you actually use a
comprehension with something that is absolutely exactly a function
call on the element in question?

> But the
> extra cost is effectively just a multiplier. It isn't going to change
> the "Big Oh" behaviour.

Sure it doesn't. In each case, the cost is linear. But the work is
linear, so I would expect the time to be linear.

> So map() here is less than a factor of two slower. I wouldn't call
> that "especially bad" -- often times, a factor of two is not important.
> What really hurts is O(N**2) performance, or worse.
>
> So at worst, map() is maybe half as fast as a list comprehension, and at
> best, its perhaps a smidgen faster. I would expect around the same
> performance, differing only by a small multiplicative factor: I wouldn't
> expect one to be thousands or even tens of times slower that the other.

But this conclusion I agree with. There is a performance difference,
but it is not overly significant. Compared to the *actual work*
involved in the task (going through one list and doing some operation
on each operation), the difference between map and a comprehension is
generally going to be negligible.

ChrisA


From bgailer at gmail.com  Thu Nov  3 17:35:17 2016
From: bgailer at gmail.com (Bob Gailer)
Date: Thu, 3 Nov 2016 17:35:17 -0400
Subject: data interpolation
In-Reply-To: <2db289bc-274a-4e05-8d4a-07822ea97b24@googlegroups.com>
References: <2db289bc-274a-4e05-8d4a-07822ea97b24@googlegroups.com>
Message-ID: <CAP1rxO5K-sa9MUQ=taBvEQL4pkdCqQsueUWwWxasaRivOzBxHg@mail.gmail.com>

On Nov 3, 2016 6:10 AM, "Heli" <hemla21 at gmail.com> wrote:
>
> Hi,
>
> I have a question about data interpolation using python. I have a big
ascii file containg data in the following format and around 200M points.
>
> id, xcoordinate, ycoordinate, zcoordinate
>
> then I have a second file containing data in the following format, ( 2M
values)
>
> id, xcoordinate, ycoordinate, zcoordinate, value1, value2, value3,...,
valueN
>
> I would need to get values for x,y,z coordinates of file 1 from values of
file2.
>
Apologies but I have no idea what you're asking. Can you give us some
examples?

> I don?t know whether my data in file1 and 2 is from structured or
unstructured grid source. I was wondering which interpolation module either
from scipy or scikit-learn you recommend me to use?
>
> I would also appreciate if you could recommend me some sample
example/reference.
>
> Thanks in Advance for your help,
> --
> https://mail.python.org/mailman/listinfo/python-list


From HooDunnit at didly42KahZidly.net  Thu Nov  3 17:52:31 2016
From: HooDunnit at didly42KahZidly.net (Cousin Stanley)
Date: Thu, 03 Nov 2016 14:52:31 -0700
Subject: Python Dice Game/Need help with my script/looping!
References: <e69bcfb9-0ee3-4570-843a-130caa16b4aa@googlegroups.com>
Message-ID: <nvgbi6$16a$1@dont-email.me>

Constantin Sorin wrote:

> Hello,I recently started to make a dice game in python.
> 
> Everything was nice and beautiful,until now.
> 
> My problem is that when I try to play and I win or lost 
> or it's equal next time it will continue only with that.
> .... 

  Following is a link to a version of your code
  rewritten to run using python3 .... 

    http://csphx.net/fire_dice.py.txt

  The only significant differences 
  between python2 and python3
  in your code are .... 

    python2 ......... python3

     print ........... print( )

     raw_input( ) ... input( )


Bob Gailer wrote: 

> ....
> The proper way to handle this 
> is to put the entire body of game() 
> in a while loop.
> ....  
> Since the values of e and f are not changed in the loop 
> he will continue to get the same thing.
> .... 

  These changes are the key
  to making the program loop
  as desired .... 

  All other changes are mostly cosmetic .... 

 
  * Note *

    I  am  partial to white space 
    both  horizontal  and  vertical ....... :-)
  

-- 
Stanley C. Kitching
Human Being
Phoenix, Arizona



From no.email at nospam.invalid  Thu Nov  3 18:35:35 2016
From: no.email at nospam.invalid (Paul Rubin)
Date: Thu, 03 Nov 2016 15:35:35 -0700
Subject: Pre-pep discussion material: in-place equivalents to map and
 filter
References: <8a268e42-87c4-4054-bd19-1dc417dd6d01@googlegroups.com>
Message-ID: <8737j8jio8.fsf@nightsong.com>

arthurhavlicek at gmail.com writes:
> I would gladly appreciate your returns on this, regarding:
> 1 - Whether a similar proposition has been made
> 2 - If you find this of any interest at all
> 3 - If you have a suggestion for improving the proposal

Bleccch.  Might be ok as a behind-the-scenes optimization by the
compiler.  If you want something like C++ move semantics, use C++.


From jladasky at itu.edu  Thu Nov  3 18:47:29 2016
From: jladasky at itu.edu (jladasky at itu.edu)
Date: Thu, 3 Nov 2016 15:47:29 -0700 (PDT)
Subject: need some kind of "coherence index" for a group of strings
In-Reply-To: <mailman.25.1478203765.7975.python-list@python.org>
References: <nvfnvu$1bc1$1@gioia.aioe.org>
 <b926915c-7862-4fe3-a74b-ae11048e2390@googlegroups.com>
 <nvg5g0$t7n$1@blaine.gmane.org>
 <mailman.25.1478203765.7975.python-list@python.org>
Message-ID: <5fe48cf9-790f-4924-80c2-0cfc41af42ae@googlegroups.com>

On Thursday, November 3, 2016 at 1:09:48 PM UTC-7, Neil D. Cerutti wrote:
> you may also be 
> able to use some items "off the shelf" from Python's difflib.

I wasn't aware of that module, thanks for the tip!

difflib.SequenceMatcher.ratio() returns a numerical value which represents the "similarity" between two strings.  I don't see a precise definition of "similar", but it may do what the OP needs.



From fillmore_remove at hotmail.com  Thu Nov  3 19:14:40 2016
From: fillmore_remove at hotmail.com (Fillmore)
Date: Thu, 3 Nov 2016 19:14:40 -0400
Subject: need some kind of "coherence index" for a group of strings
References: <nvfnvu$1bc1$1@gioia.aioe.org>
 <b926915c-7862-4fe3-a74b-ae11048e2390@googlegroups.com>
 <nvg5g0$t7n$1@blaine.gmane.org>
 <mailman.25.1478203765.7975.python-list@python.org>
 <5fe48cf9-790f-4924-80c2-0cfc41af42ae@googlegroups.com>
Message-ID: <nvggcv$lc2$2@gioia.aioe.org>

On 11/3/2016 6:47 PM, jladasky at itu.edu wrote:
> On Thursday, November 3, 2016 at 1:09:48 PM UTC-7, Neil D. Cerutti wrote:
>> you may also be
>> able to use some items "off the shelf" from Python's difflib.
>
> I wasn't aware of that module, thanks for the tip!
>
> difflib.SequenceMatcher.ratio() returns a numerical value which represents
 > the "similarity" between two strings.  I don't see a precise 
definition of
 > "similar", but it may do what the OP needs.
>


I may end up rolling my own algo, but thanks for the tip, this does seem 
like useful stuff indeed




From jladasky at itu.edu  Thu Nov  3 19:14:53 2016
From: jladasky at itu.edu (jladasky at itu.edu)
Date: Thu, 3 Nov 2016 16:14:53 -0700 (PDT)
Subject: need some kind of "coherence index" for a group of strings
In-Reply-To: <5fe48cf9-790f-4924-80c2-0cfc41af42ae@googlegroups.com>
References: <nvfnvu$1bc1$1@gioia.aioe.org>
 <b926915c-7862-4fe3-a74b-ae11048e2390@googlegroups.com>
 <nvg5g0$t7n$1@blaine.gmane.org>
 <mailman.25.1478203765.7975.python-list@python.org>
 <5fe48cf9-790f-4924-80c2-0cfc41af42ae@googlegroups.com>
Message-ID: <38742263-0fc6-4270-9eb7-9bcc58462c60@googlegroups.com>

On Thursday, November 3, 2016 at 3:47:41 PM UTC-7, jlad... at itu.edu wrote:
> On Thursday, November 3, 2016 at 1:09:48 PM UTC-7, Neil D. Cerutti wrote:
> > you may also be 
> > able to use some items "off the shelf" from Python's difflib.
> 
> I wasn't aware of that module, thanks for the tip!
> 
> difflib.SequenceMatcher.ratio() returns a numerical value which represents the "similarity" between two strings.  I don't see a precise definition of "similar", but it may do what the OP needs.

Following up to myself... I just experimented with difflib.SequenceMatcher.ratio() and discovered something.  The algorithm is not "commutative."  That is, it doesn't ALWAYS produce the same ratio when the two strings are swapped.

Here's an excerpt from my interpreter session.

==========

In [1]: from difflib import SequenceMatcher

In [2]: import numpy as np

In [3]: sim = np.zeros((4,4))


== snip ==


In [10]: strings
Out[10]: 
('Here is a string.',
 'Here is a slightly different string.',
 'This string should be significantly different from the other two?',
 "Let's look at all these string similarity values in a matrix.")

In [11]: for r, s1 in enumerate(strings):
   ....:     for c, s2 in enumerate(strings):
   ....:         m = SequenceMatcher(lambda x:x=="", s1, s2)
   ....:         sim[r,c] = m.ratio()
   ....:

In [12]: sim
Out[12]: 
array([[ 1.        ,  0.64150943,  0.2195122 ,  0.30769231],
       [ 0.64150943,  1.        ,  0.47524752,  0.30927835],
       [ 0.2195122 ,  0.45544554,  1.        ,  0.28571429],
       [ 0.30769231,  0.28865979,  0.33333333,  1.        ]])

==========

The values along the matrix diagonal, of course, are all ones, because each string was compared to itself.

I also expected the values reflected across the matrix diagonal to match.  The first row does in fact match the first column.  The remaining numbers disagree somewhat.  The differences are not large, but they are there.  I don't know the reason why.  Caveat programmer.


From tjreedy at udel.edu  Thu Nov  3 20:03:42 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 3 Nov 2016 20:03:42 -0400
Subject: Pre-pep discussion material: in-place equivalents to map and
 filter
In-Reply-To: <8a268e42-87c4-4054-bd19-1dc417dd6d01@googlegroups.com>
References: <8a268e42-87c4-4054-bd19-1dc417dd6d01@googlegroups.com>
Message-ID: <nvgj8t$sm6$1@blaine.gmane.org>

On 11/3/2016 2:56 AM, arthurhavlicek at gmail.com wrote:

> lst = [ item for item in lst if predicate(item) ]
> lst = [ f(item) for item in lst ]
>
> Both these expressions feature redundancy, lst occurs twice and item at least twice. Additionally, the readability is hurt, because one has to dive through the semantics of the comprehension to truely understand I am filtering the list or remapping its values.
...
> A language support for these operations to be made in-place could improve the efficiency of this operations through reduced use of memory.

We already have that: slice assignment with an iterator.

lst[:] = (item for item in list if predicate(item))
lst[:] = map(f, lst)  # iterator in 3.x.

To save memory, stop using unneeded temporary lists and use iterators 
instead.  If slice assignment is done as I hope it will optimize remain 
memory operations.  (But I have not read the code.) It should overwrite 
existing slots until either a) the iterator is exhausted or b) existing 
memory is used up.  When lst is both source and destination, only case 
a) can happen.  When it does, the list can be finalized with its new 
contents.

As for timings.

from timeit import Timer
setup = """data = list(range(10000))
def func(x):
     return x
"""
t1a = Timer('data[:] = [func(a) for a in data]', setup=setup)
t1b = Timer('data[:] = (func(a) for a in data)', setup=setup)
t2a = Timer('data[:] = list(map(func, data))', setup=setup)
t2b = Timer('data[:] = map(func, data)', setup=setup)

print('t1a', min(t1a.repeat(number=500, repeat=7)))
print('t1b', min(t1b.repeat(number=500, repeat=7)))
print('t2a', min(t2a.repeat(number=500, repeat=7)))
print('t2b', min(t2b.repeat(number=500, repeat=7)))
#
t1a 0.5675313005414555
t1b 0.7034254675598604
t2a 0.5181285985208888
t2b 0.5196112759726024

If f does more work, the % difference among these will decrease.


-- 
Terry Jan Reedy



From nimbiotics at gmail.com  Thu Nov  3 22:37:02 2016
From: nimbiotics at gmail.com (Mario R. Osorio)
Date: Thu, 3 Nov 2016 19:37:02 -0700 (PDT)
Subject: need some kind of "coherence index" for a group of strings
In-Reply-To: <nvfnvu$1bc1$1@gioia.aioe.org>
References: <nvfnvu$1bc1$1@gioia.aioe.org>
Message-ID: <ffe2b193-51aa-4830-8f94-ec8e89e8b46d@googlegroups.com>

I don't know much about these topics but, wouldn't soundex do the job??

 On Thursday, November 3, 2016 at 12:18:19 PM UTC-4, Fillmore wrote:
> Hi there, apologies for the generic question. Here is my problem let's 
> say that I have a list of lists of strings.
> 
> list1:    #strings are sort of similar to one another
> 
>    my_nice_string_blabla
>    my_nice_string_blqbli
>    my_nice_string_bl0bla
>    my_nice_string_aru
> 
> 
> list2:    #strings are mostly different from one another
> 
>    my_nice_string_blabla
>    some_other_string
>    yet_another_unrelated string
>    wow_totally_different_from_others_too
> 
> 
> I would like an algorithm that can look at the strings and determine 
> that strings in list1 are sort of similar to one another, while the 
> strings in list2 are all different.
> Ideally, it would be nice to have some kind of 'coherence index' that I 
> can exploit to separate lists given a certain threshold.
> 
> I was about to concoct something using levensthein distance, but then I 
> figured that it would be expensive to compute and I may be reinventing 
> the wheel.
> 
> Thanks in advance to python masters that may have suggestions...



From rustompmody at gmail.com  Fri Nov  4 01:27:03 2016
From: rustompmody at gmail.com (Rustom Mody)
Date: Thu, 3 Nov 2016 22:27:03 -0700 (PDT)
Subject: advanced SimpleHTTPServer?
In-Reply-To: <mailman.7.1478116364.7975.python-list@python.org>
References: <nvc4g9$l6i$1@news2.informatik.uni-stuttgart.de>
 <CAMw+j7Jes=L+BMQ2QymfxF7OWKTdOtdbCFzCRk1N6X7vB5unPw@mail.gmail.com>
 <6d205aab-dd5e-30bd-36dd-6c7fb3b0dfd3@harvee.org>
 <CAMw+j7JSZhLhtV_rgVSdPq5rKqsp_EgnwCgn-fSVnwQZtfYPCw@mail.gmail.com>
 <4079c7a3-074f-e070-08be-e0168a72dfc4@harvee.org>
 <mailman.7.1478116364.7975.python-list@python.org>
Message-ID: <0cd13a77-7de4-42f1-96e1-b666889a878f@googlegroups.com>

On Thursday, November 3, 2016 at 1:23:05 AM UTC+5:30, Eric S. Johansson wrote:
> On 11/2/2016 2:40 PM, Chris Warrick wrote:
> > Because, as the old saying goes, any sufficiently complicated Bottle
> > or Flask app contains an ad hoc, informally-specified, bug-ridden,
> > slow implementation of half of Django. (In the form of various plugins
> > to do databases, accounts, admin panels etc.)
> 
> That's not a special attribute of bottle, flask or Django. Ad hoc,
> informally specified, bug ridden slow implementations abound.  We focus
> too much on scaling up and not enough on scaling down. We (designers) 
> also have not properly addressed configuration complexity issues.

This scaling up vs down idea is an important one.
Related to Buchberger?s blackbox whitebox principle
 
> 
> If I'm going do something once, if it cost me more than a couple of
> hours to figure it out, it's too expensive in general but definitely if
> I forget what I learned. That's why bottle/flask systems meet and need.
> They're not too expensive to forget what you learned.
> 
> Django makes the cost of forgetting extremely expensive. I think of
> using Django as career  rather than a toolbox.

Thats snide... and probably accurate ;-)
Among my more unpleasant programming experiences was Ruby-on-Rails
And my impression is that Ruby is fine; Rails not
Django I dont know and my impression is its a shade better than Rails

It would be nice to discover the bottle inside the flask inside django

Put differently:
Frameworks are full-featured and horrible to use
APIs are elegant but ultimately underpowered
DSLs (eg requests) are in intermediate sweetspot; we need more DSL-families


From walters.justin01 at gmail.com  Fri Nov  4 01:38:56 2016
From: walters.justin01 at gmail.com (justin walters)
Date: Thu, 3 Nov 2016 22:38:56 -0700
Subject: advanced SimpleHTTPServer?
In-Reply-To: <0cd13a77-7de4-42f1-96e1-b666889a878f@googlegroups.com>
References: <nvc4g9$l6i$1@news2.informatik.uni-stuttgart.de>
 <CAMw+j7Jes=L+BMQ2QymfxF7OWKTdOtdbCFzCRk1N6X7vB5unPw@mail.gmail.com>
 <6d205aab-dd5e-30bd-36dd-6c7fb3b0dfd3@harvee.org>
 <CAMw+j7JSZhLhtV_rgVSdPq5rKqsp_EgnwCgn-fSVnwQZtfYPCw@mail.gmail.com>
 <4079c7a3-074f-e070-08be-e0168a72dfc4@harvee.org>
 <mailman.7.1478116364.7975.python-list@python.org>
 <0cd13a77-7de4-42f1-96e1-b666889a878f@googlegroups.com>
Message-ID: <CAO1D73EHokZAegEkUnR0HM-mBv7EjUKR_WQk7+Yi5hRpx40_zQ@mail.gmail.com>

On Thu, Nov 3, 2016 at 10:27 PM, Rustom Mody <rustompmody at gmail.com> wrote:

> On Thursday, November 3, 2016 at 1:23:05 AM UTC+5:30, Eric S. Johansson
> wrote:
> > On 11/2/2016 2:40 PM, Chris Warrick wrote:
> > > Because, as the old saying goes, any sufficiently complicated Bottle
> > > or Flask app contains an ad hoc, informally-specified, bug-ridden,
> > > slow implementation of half of Django. (In the form of various plugins
> > > to do databases, accounts, admin panels etc.)
> >
> > That's not a special attribute of bottle, flask or Django. Ad hoc,
> > informally specified, bug ridden slow implementations abound.  We focus
> > too much on scaling up and not enough on scaling down. We (designers)
> > also have not properly addressed configuration complexity issues.
>
> This scaling up vs down idea is an important one.
> Related to Buchberger?s blackbox whitebox principle
>
> >
> > If I'm going do something once, if it cost me more than a couple of
> > hours to figure it out, it's too expensive in general but definitely if
> > I forget what I learned. That's why bottle/flask systems meet and need.
> > They're not too expensive to forget what you learned.
> >
> > Django makes the cost of forgetting extremely expensive. I think of
> > using Django as career  rather than a toolbox.
>
> Thats snide... and probably accurate ;-)
> Among my more unpleasant programming experiences was Ruby-on-Rails
> And my impression is that Ruby is fine; Rails not
> Django I dont know and my impression is its a shade better than Rails
>
> It would be nice to discover the bottle inside the flask inside django
>
> Put differently:
> Frameworks are full-featured and horrible to use
> APIs are elegant but ultimately underpowered
> DSLs (eg requests) are in intermediate sweetspot; we need more DSL-families
> --
> https://mail.python.org/mailman/listinfo/python-list
>

I work with Django every day. Knowing Django is like knowing another
ecosystem. It's totally
worth learning though. The speed of development is absolutely unbeatable. I
can build a fully featured
and good-looking blog in about 10 minutes. It's nuts.

The best part about it though, is that it's really just simple Python under
the hood for the most part. You
can override or modify any part of it to make it work in exactly the way
you want it to. I'm a huge Django fanboy,
so excuse the gushing. The docs are also some of the most comprehensive
I've ever seen.


From duncan at invalid.invalid  Fri Nov  4 10:07:07 2016
From: duncan at invalid.invalid (duncan smith)
Date: Fri, 4 Nov 2016 14:07:07 +0000
Subject: need some kind of "coherence index" for a group of strings
In-Reply-To: <nvfnvu$1bc1$1@gioia.aioe.org>
References: <nvfnvu$1bc1$1@gioia.aioe.org>
Message-ID: <gC0Tz.24263$vF2.16237@fx43.iad>

On 03/11/16 16:18, Fillmore wrote:
> 
> Hi there, apologies for the generic question. Here is my problem let's
> say that I have a list of lists of strings.
> 
> list1:    #strings are sort of similar to one another
> 
>   my_nice_string_blabla
>   my_nice_string_blqbli
>   my_nice_string_bl0bla
>   my_nice_string_aru
> 
> 
> list2:    #strings are mostly different from one another
> 
>   my_nice_string_blabla
>   some_other_string
>   yet_another_unrelated string
>   wow_totally_different_from_others_too
> 
> 
> I would like an algorithm that can look at the strings and determine
> that strings in list1 are sort of similar to one another, while the
> strings in list2 are all different.
> Ideally, it would be nice to have some kind of 'coherence index' that I
> can exploit to separate lists given a certain threshold.
> 
> I was about to concoct something using levensthein distance, but then I
> figured that it would be expensive to compute and I may be reinventing
> the wheel.
> 
> Thanks in advance to python masters that may have suggestions...
> 
> 
> 

https://pypi.python.org/pypi/jellyfish/

Duncan


From marco.nawijn at colosso.nl  Fri Nov  4 11:07:53 2016
From: marco.nawijn at colosso.nl (marco.nawijn at colosso.nl)
Date: Fri, 4 Nov 2016 08:07:53 -0700 (PDT)
Subject: data interpolation
In-Reply-To: <2db289bc-274a-4e05-8d4a-07822ea97b24@googlegroups.com>
References: <2db289bc-274a-4e05-8d4a-07822ea97b24@googlegroups.com>
Message-ID: <3e663994-8f74-4f07-91c1-a41e0eda712b@googlegroups.com>

On Thursday, November 3, 2016 at 11:08:34 AM UTC+1, Heli wrote:
> Hi, 
> 
> I have a question about data interpolation using python. I have a big ascii file containg data in the following format and around 200M points. 
> 
> id, xcoordinate, ycoordinate, zcoordinate
> 
> then I have a second file containing data in the following format, ( 2M values) 
> 
> id, xcoordinate, ycoordinate, zcoordinate, value1, value2, value3,..., valueN
> 
> I would need to get values for x,y,z coordinates of file 1 from values of file2.  
> 
> I don?t know whether my data in file1 and 2 is from structured or unstructured grid source. I was wondering which interpolation module either from scipy or scikit-learn you recommend me to use?
> 
> I would also appreciate if you could recommend me some sample example/reference. 
> 
> Thanks in Advance for your help,

Take a look at the scipy.spatial.KDTree class:
https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.spatial.KDTree.html

Given your example, you would build the tree (using the coordinates) 
from the second file. Subsequently, you can use one of the query 
methods for every point in your first file. From this it is up to 
you how to transfer (interpolate) the values.

Marco




From arthurhavlicek at gmail.com  Fri Nov  4 14:21:09 2016
From: arthurhavlicek at gmail.com (Arthur Havlicek)
Date: Fri, 4 Nov 2016 19:21:09 +0100
Subject: Pre-pep discussion material: in-place equivalents to map and
 filter
In-Reply-To: <nvgj8t$sm6$1@blaine.gmane.org>
References: <8a268e42-87c4-4054-bd19-1dc417dd6d01@googlegroups.com>
 <nvgj8t$sm6$1@blaine.gmane.org>
Message-ID: <CAPpswAK7NTSbi3-bdw--6oeJ8rhYT99_zUQtztAAWmV0p+tBjA@mail.gmail.com>

> If slice assignment is done as I hope it will optimize remain memory
operations.

Bad news.

http://stackoverflow.com/questions/4948293/python-slice-assignment-memory-usage/4948508#4948508

> If you want something like C++ move semantics, use C++.

I don't see anything like this in my proposal. If any in-place operation is
"C++ semantics" how do you explain there is already a bunch of in-place
operator stuffed in Python that have even less justification to their
existance than map or filter does such as sort/sorted, reverse/reversed ?
Especially troubling since the optimisation in a sort operation is likely
to be less significant than in a linear algorithm.

2016-11-04 1:03 GMT+01:00 Terry Reedy <tjreedy at udel.edu>:

> On 11/3/2016 2:56 AM, arthurhavlicek at gmail.com wrote:
>
> lst = [ item for item in lst if predicate(item) ]
>> lst = [ f(item) for item in lst ]
>>
>> Both these expressions feature redundancy, lst occurs twice and item at
>> least twice. Additionally, the readability is hurt, because one has to dive
>> through the semantics of the comprehension to truely understand I am
>> filtering the list or remapping its values.
>>
> ...
>
>> A language support for these operations to be made in-place could improve
>> the efficiency of this operations through reduced use of memory.
>>
>
> We already have that: slice assignment with an iterator.
>
> lst[:] = (item for item in list if predicate(item))
> lst[:] = map(f, lst)  # iterator in 3.x.
>
> To save memory, stop using unneeded temporary lists and use iterators
> instead.  If slice assignment is done as I hope it will optimize remain
> memory operations.  (But I have not read the code.) It should overwrite
> existing slots until either a) the iterator is exhausted or b) existing
> memory is used up.  When lst is both source and destination, only case a)
> can happen.  When it does, the list can be finalized with its new contents.
>
> As for timings.
>
> from timeit import Timer
> setup = """data = list(range(10000))
> def func(x):
>     return x
> """
> t1a = Timer('data[:] = [func(a) for a in data]', setup=setup)
> t1b = Timer('data[:] = (func(a) for a in data)', setup=setup)
> t2a = Timer('data[:] = list(map(func, data))', setup=setup)
> t2b = Timer('data[:] = map(func, data)', setup=setup)
>
> print('t1a', min(t1a.repeat(number=500, repeat=7)))
> print('t1b', min(t1b.repeat(number=500, repeat=7)))
> print('t2a', min(t2a.repeat(number=500, repeat=7)))
> print('t2b', min(t2b.repeat(number=500, repeat=7)))
> #
> t1a 0.5675313005414555
> t1b 0.7034254675598604
> t2a 0.5181285985208888
> t2b 0.5196112759726024
>
> If f does more work, the % difference among these will decrease.
>
>
>
> --
> Terry Jan Reedy
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From steve+python at pearwood.info  Fri Nov  4 20:42:48 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sat, 05 Nov 2016 11:42:48 +1100
Subject: Pre-pep discussion material: in-place equivalents to map and
 filter
References: <8a268e42-87c4-4054-bd19-1dc417dd6d01@googlegroups.com>
 <581af57a$0$1585$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmrQdo2J2ztRz-waim2rsaugi-CJDzOrFe441-OqgoS9Pg@mail.gmail.com>
 <mailman.17.1478181910.7975.python-list@python.org>
 <581b6d3a$0$1583$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmp9Sr1=-FoOz=aqVxXaFQ3AjZWUH4jq868nJEFdhbH4yA@mail.gmail.com>
 <mailman.29.1478208878.7975.python-list@python.org>
Message-ID: <581d2b0b$0$1606$c3e8da3$5496439d@news.astraweb.com>

On Fri, 4 Nov 2016 08:34 am, Chris Angelico wrote:

[...]
> List comps themselves involve one function call (zero in Py2). What
> you do inside the expression is your business. Do you agree that list
> comps don't have the overhead of opening and closing files?

/tongue firmly in cheek

I'd like to see you run a Python script containing a list comprehension
without opening and closing the .py file.

:-P


Okay, I see what you are getting at now: in CPython 3, list comprehensions
are implemented in such a way that the list comprehension requires a
minimum of one function call, while list(map(func, iterable))) requires a
minimum of O(N)+2 function calls: a call to list, a call to map, and a call
to func for each of N values.

That's *technically* true, but it is an implementation detail. I'm sure that
some day PyPy or Nuitka or maybe even CPython itself will start in-lining
at least some functions (if PyPy doesn't already do so). That would be an
obvious optimization to apply to map() and filter().

As far as the speed of map() versus list comps, my micro-benchmarks show
that at least on my computer, map() can be marginally but consistently
faster than a list comp once you equalise that cost of function calls, that
is, if the list comp explicitly calls a function. I don't think that speed
difference is significant, so let's just call them "equally fast" when
comparing similar cases:

    [func(obj) for obj in iterable]

    map(func, iterable)


But of course you're right that list comprehensions give you the opportunity
to avoid that function call -- at least in CPython. I already agreed with
that, but to emphasise what I've already agreed, I'll say it again :-)

If you can manually in-line func() as a single expression inside the list
comprehension:

    [(spam + len(obj.eggs))*2 for obj in iterable]

then you can expect to save the cost of N function calls, which may be
significant. As I said earlier, that's why we have list comps.

(But on the other hand, if the expression is expensive enough, the cost of
an extra function call may be utterly insignificant.)

The point that I am trying to make is that none of these facts justifies the
claim that map() performs "especially bad" compared to list comprehensions.
According to my tests, *at worst* map() will be a bit better than half as
fast as a list comprehension, and at best just as fast if not slightly
faster.


>> Here's some timing results using 3.5 on my computer. For simplicity, so
>> folks can replicate the test themselves, here's the timing code:
>>
>>
>> from timeit import Timer
>> setup = """data = list(range(10000))
>> def func(x):  # simulate some calculation
>>     return {x+1: x**2}
>> """
>> t1 = Timer('[func(a) for a in data]', setup=setup)
>> t2 = Timer('list(map(func, data))', setup=setup)
> 
> This is very different from the original example, about which the OP
> said that map performs badly, and you doubted it.

I didn't read the OP as making a specific claim about these two *specific*
map and filter examples:

    lst = map (lambda x: x*5, lst)
    lst = filter (lambda x: x%3 == 1, lst)

I read these as mere examples of a general claim that map and
filter "perform especially bad in CPython compared to a comprehension".

But just for the exercise, I repeated my benchmarks with these specific
examples, comparing:

    list(map(lambda x: x*5, data))
    [x*5 for x in data]

and 

    list(filter(lambda x: x%3 == 1, data))
    [x for x in data if x%3 == 1]

and again got a roughly factor of two performance difference, with the list
comp being faster. I don't think that justifies the claim of "especially
bad", which to me implies something much worse. If you're going to describe
a mere factor of two as "especially bad", what words do we have left for
something that is ten thousand times slower?

As the wisest man in the universe once said, hyperbole is the most terrible,
awful crime against humanity.

*wink*


[...]
> Thing is, this is extremely common. How often do you actually use a
> comprehension with something that is absolutely exactly a function
> call on the element in question?

"This" being something that can be in-lined in the body of the list comp.

Sure. I cheerfully acknowledge that list comps where you can write an
in-line expression are very common. That's the beauty of list comps!


[...]
> But this conclusion I agree with. There is a performance difference,
> but it is not overly significant. Compared to the *actual work*
> involved in the task (going through one list and doing some operation
> on each operation), the difference between map and a comprehension is
> generally going to be negligible.

I wouldn't go quite so far as to say "negligible" -- a factor of two speed
up on a large list is not something to be sneezed at. But I think we're
converging on agreement: list comps and map/filter typically have
comparable performance, and as the cost of the work done increases, the
extra overhead of a function call becomes less and less significant.



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From rosuav at gmail.com  Fri Nov  4 21:16:16 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 5 Nov 2016 12:16:16 +1100
Subject: Pre-pep discussion material: in-place equivalents to map and
 filter
In-Reply-To: <581d2b0b$0$1606$c3e8da3$5496439d@news.astraweb.com>
References: <8a268e42-87c4-4054-bd19-1dc417dd6d01@googlegroups.com>
 <581af57a$0$1585$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmrQdo2J2ztRz-waim2rsaugi-CJDzOrFe441-OqgoS9Pg@mail.gmail.com>
 <mailman.17.1478181910.7975.python-list@python.org>
 <581b6d3a$0$1583$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmp9Sr1=-FoOz=aqVxXaFQ3AjZWUH4jq868nJEFdhbH4yA@mail.gmail.com>
 <mailman.29.1478208878.7975.python-list@python.org>
 <581d2b0b$0$1606$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPTjJmohS9yu5cNYPCs4eQ0uCuS93j-4sk7nEAbV8z8huosU3g@mail.gmail.com>

On Sat, Nov 5, 2016 at 11:42 AM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> On Fri, 4 Nov 2016 08:34 am, Chris Angelico wrote:
>
> [...]
>> List comps themselves involve one function call (zero in Py2). What
>> you do inside the expression is your business. Do you agree that list
>> comps don't have the overhead of opening and closing files?
>
> /tongue firmly in cheek
>
> I'd like to see you run a Python script containing a list comprehension
> without opening and closing the .py file.
>
> :-P

You got me! Let's call it a night.
-- Kristoff (nearly my namesake)

> Okay, I see what you are getting at now: in CPython 3, list comprehensions
> are implemented in such a way that the list comprehension requires a
> minimum of one function call, while list(map(func, iterable))) requires a
> minimum of O(N)+2 function calls: a call to list, a call to map, and a call
> to func for each of N values.
>
> That's *technically* true, but it is an implementation detail. I'm sure that
> some day PyPy or Nuitka or maybe even CPython itself will start in-lining
> at least some functions (if PyPy doesn't already do so). That would be an
> obvious optimization to apply to map() and filter().

Mmmm, interesting. The fact still remains that map depends on some
kind of "object representation" of a block of code (since it's being
passed to some other function), where a comprehension can be
implemented as an actual expression. So either Python-the-language
needs a way to pass around lightweight blocks of code, or the
interpreter (for some instance of 'the interpreter') needs to
recognize the function calls and optimize them away, or list comps
will always have an inherent advantage over map.

> As far as the speed of map() versus list comps, my micro-benchmarks show
> that at least on my computer, map() can be marginally but consistently
> faster than a list comp once you equalise that cost of function calls, that
> is, if the list comp explicitly calls a function. I don't think that speed
> difference is significant, so let's just call them "equally fast" when
> comparing similar cases:
>
>     [func(obj) for obj in iterable]
>
>     map(func, iterable)

Right, which is why a lot of style guides recommend against the first
form, *in this specific instance*. Using map with a lambda function,
or a comprehension with nothing but a function call, is rightly called
out in code review.

> I didn't read the OP as making a specific claim about these two *specific*
> map and filter examples:
>
>     lst = map (lambda x: x*5, lst)
>     lst = filter (lambda x: x%3 == 1, lst)
>
> I read these as mere examples of a general claim that map and
> filter "perform especially bad in CPython compared to a comprehension".
> ... I don't think that justifies the claim of "especially
> bad", which to me implies something much worse. If you're going to describe
> a mere factor of two as "especially bad", what words do we have left for
> something that is ten thousand times slower?
>
> As the wisest man in the universe once said, hyperbole is the most terrible,
> awful crime against humanity.
>
> *wink*

Ah, now we get to the point where we disagreed. I was responding to a
misunderstanding of your position - I thought you disagreed that the
performance difference could even be significant, but you were arguing
against the "especially bad". Gotcha. In that case, I believe we're in
agreement; even a two-to-one difference isn't "especially bad" here,
and that would be an extreme case.

>> But this conclusion I agree with. There is a performance difference,
>> but it is not overly significant. Compared to the *actual work*
>> involved in the task (going through one list and doing some operation
>> on each operation), the difference between map and a comprehension is
>> generally going to be negligible.
>
> I wouldn't go quite so far as to say "negligible" -- a factor of two speed
> up on a large list is not something to be sneezed at. But I think we're
> converging on agreement: list comps and map/filter typically have
> comparable performance, and as the cost of the work done increases, the
> extra overhead of a function call becomes less and less significant.

I said negligible because the factor of two disappears almost
completely when you add a large constant factor to it. What's the
performance of these?

def lie(n):
    """it's not a fib, honest"""
    if n < 2: return 3
    return lie(n-1) + lie(n-2)

mapped = list(map(lie, range(50)))
comprehended = [lie(x) for x in range(50)]
badly_mapped = list(map(lambda x: lie(x), range(50)))

By any reasonable metric, I would expect all three to have extremely
comparable performance. The difference between map's best case
(passing an existing function), a list comp, and map's worst case
(wrapping the function in a useless lambda function) might be two to
one, but it's negligible compared to the surpassing cost of those
massively-nested lies. Oh, what a tangled web we weave...

ChrisA


From arthurhavlicek at gmail.com  Sat Nov  5 01:07:14 2016
From: arthurhavlicek at gmail.com (arthurhavlicek at gmail.com)
Date: Fri, 4 Nov 2016 22:07:14 -0700 (PDT)
Subject: Pre-pep discussion material: in-place equivalents to map and
 filter
In-Reply-To: <8a268e42-87c4-4054-bd19-1dc417dd6d01@googlegroups.com>
References: <8a268e42-87c4-4054-bd19-1dc417dd6d01@googlegroups.com>
Message-ID: <a0388e6e-3466-468b-92e1-5098db6e9ae7@googlegroups.com>

> I don't think that justifies the claim of "especially
> bad", which to me implies something much worse.

Quicksort has built its popularity by performing better by "a mere factor two" better than mergesort and heapsort. It became the reference sorting algorithm even though its worst case complexity is worse than its competitors.

You can insurge about the fact I call a factor two to be especially bad, and you are right it is an hyperbole in the general case, but I am still right in the specific context of a frequently used linear algorithm.


From xzldyx at hotmail.com  Sat Nov  5 02:14:50 2016
From: xzldyx at hotmail.com (xzldyx at hotmail.com)
Date: Fri, 4 Nov 2016 23:14:50 -0700 (PDT)
Subject: How to pass C++ function pointer argument in embedded python
 environment?
Message-ID: <f591a409-eea1-438e-bf5b-c23627adf89c@googlegroups.com>

Hi,All,
    background:I have a python api like this,
       def a(arg1,arg2,progress_callback = None)
    obviously argument progress_callback is a callback function.In python,I can define a function like this:
       def pro_call(prog_arg1,prog_arg2):
           #do something with arg1 & arg2
    and I just need call a function like follow and it worked
       a(arg1,arg2,progress_callback = pro_call)


    question: I don't know how to do like the former exam in embedded python environment.


    thank you for your reading ,thanks for any response!


From bob.martin at excite.com  Sat Nov  5 03:23:02 2016
From: bob.martin at excite.com (Bob Martin)
Date: Sat, 05 Nov 2016 07:23:02 GMT
Subject: Call a shell command from Python
References: <bOWdnTgDbsjFW4vFnZ2dnUU7-LPNnZ2d@giganews.com>
 <85r36x9lgi.fsf@benfinney.id.au>
 <mailman.19.1477889083.17234.python-list@python.org>
 <20255226.6Emhk5qWAg@PointedEars.de>
Message-ID: <e85fmmFhk60U1@mid.individual.net>

in 767198 20161104 142132 Thomas 'PointedEars' Lahn <PointedEars at web.de> wrote:
>Ben Finney wrote:
>
>> Note that ???sudo??? is specifically designed to be invoked interactively,
>
>Nonsense.
>
>> seeking to verify that the current user has credentials to run the
>> command.
>
>NOPASSWD is not the default in sudoers(5),

It is on the Raspberry Pi

 but conclude from that what you
>concluded is a bit far-fetched, to say the least.


From steve+python at pearwood.info  Sat Nov  5 04:42:55 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sat, 05 Nov 2016 19:42:55 +1100
Subject: Pre-pep discussion material: in-place equivalents to map and
 filter
References: <8a268e42-87c4-4054-bd19-1dc417dd6d01@googlegroups.com>
 <581af57a$0$1585$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmrQdo2J2ztRz-waim2rsaugi-CJDzOrFe441-OqgoS9Pg@mail.gmail.com>
 <mailman.17.1478181910.7975.python-list@python.org>
 <581b6d3a$0$1583$c3e8da3$5496439d@news.astraweb.com>
 <CAPpswAJF7PzVb1nyqD0fj=2SuW7pT6QNrH_vz1+CVSNGU-Pv2Q@mail.gmail.com>
 <mailman.26.1478204949.7975.python-list@python.org>
Message-ID: <581d9b90$0$1599$c3e8da3$5496439d@news.astraweb.com>

I've been giving your proposal a bit more thought, and while I can't say I'm
really keep on the idea, I have warmed slightly to it.


On Fri, 4 Nov 2016 07:29 am, Arthur Havlicek wrote:

> I understand that, the cost of change is such that it's very unlikely
> something like this ever goes into Python, but I feel like the interest of
> the proposition is being underestimated here, that's why I'm going to
> argue a few points and give a bit more context as needed.
> 
>> While mapping and filtering are common operations, I'm not sure mapping
>> and filtering and then reassigning back to the original sequence is
>> especially common.
> 
> It depends of your context. On the last 3 months, I stumbled across this
> at least 3 times,

I don't know who you are quoting there. It is considered impolite to quote
people without giving attribution, and makes it harder to respond. But for
what it's worth, I agree with this person.

In my code, it is quite common for me to map back to the same *variable*,
for example I might write something like this:

def process(alist):
    alist = [float(x) for x in alist]
    ...


But that is not the same as *in-place* modification of the list. I would
very rarely modify the list in place, but if I did, I would write it like
this:

    alist[:] = [float(x) for x in alist]


In my experience, wanting to modify a list in-place without creating a
temporary version first is unusual. And *needing* to do so (rather than
doing so as premature optimization) is even rarer.

That's my experience. But if you can demonstrate the need for in-place
modifications, that changes the situation somewhat.


> which is 3 times more than I used a lambda or a
> metaclass or a decorator or other fancy language feature that we simply
> avoid whenever possible.

You aren't helping your case by describing "lambda or ... decorator" as
fancy language features to be avoided.

The impression that gives (hopefully this is the wrong impression!) is that
you are a barely competent Python developer, fearful and suspicious of some
rather simple, powerful and widely-used features, stuck in an ancient 1980s
programming paradigm.

I trust that isn't actually the case, but by describing decorators and
lambda as too fancy to use, you're giving a good impression of somebody who
doesn't know what they're doing.


[...]
> The reason I'm being especially impacted by this is because I am
> maintainer of a decent-sized Python application (~50-100K lines of code)
> that extensively uses lists and dictionaries. We value "low level" data
> manipulation and efficiency a lot more than complex, non-obvious
> constructs.

And what do you consider "complex, non-obvious"?


[...]
> Like most Python programmers, I'm not in the case of needing a performance
> boost very bad, but that does not mean I disregard performance entirely.
> The need of performance is not so binary that it either don't matter at
> all or is enough to motivate a rewrite.

Indeed.

But you're arguing for a new feature on the basis that it will boost
performance, without giving any reason to think that it actually will lead
to a performance improvement!

I know that there is a chicken-and-egg problem here. Nobody wants to spend
time writing a new feature if there's no possibly chance for it to be
accepted, but when your sole argument for a new feature is that it will be
more efficient (in both memory and time!), then you need to prove that is
the case!

In other words, in my opinion:

- you would need to prove that there is a widespread need for in-place
modification of lists, where the lists are big enough that using a
temporary list is not practical;

- you would have to demonstrate a good reason to think that this new feature
will actually be more efficient and faster

before this feature would be seriously considered for addition to the
language.

You might think that it is obvious that in-place modification MUST be faster
since it avoids making a temporary copy, but that's not obviously true at
all! Not for a high-level language like Python. Its often the case that
algorithms can exchange space for time (use more memory to save time, or
use more time to save memory). It may be that this is one of the times.

Even when doing (nearly) everything in pure Python, making a new list and
then doing a slice assignment is virtually as fast as writing directly to
the original list:

py> from timeit import Timer
py> setup = "data = list(range(10000))"
py> t1 = Timer("""for i, a in enumerate(data):
...     data[i] = a+1
... """, setup=setup)
py> 
py> t2 = Timer("""tmp = [None]*len(data)
... for i, a in enumerate(data):
...     tmp[i] = a+1
... data[:] = tmp
... """, setup=setup)
py>
py> min(t1.repeat(number=10000, repeat=7))
36.63875983003527
py>
py> min(t2.repeat(number=10000, repeat=7))
37.70047474466264


Taking the plain Python for-loop, modifying the list directly in place, we
see that making a temporary list and then copying it over the original list
is just 3% slower. (And that's probably just random noise.)


But moving the heavy work into a list comprehension before copying over the
original: 

py> t3 = Timer("data[:] = [a+1 for a in data]", setup=setup)
py> min(t3.repeat(number=10000, repeat=7))
21.755106460303068

is 40% faster! Based on these results, my conclusion is that there's no
speed advantage to writing directly into the list, compared to writing to a
temporary list and then copying the lot in one go.

(I find these results unintuitive, but I've seen them often enough that I'm
confident they are real.)



> About Readability & Redundancy
> 
> I have misused the terms here, but I wasn't expecting so much nitpicking.

You must be new here :-)





-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From jon+usenet at unequivocal.eu  Sat Nov  5 05:45:12 2016
From: jon+usenet at unequivocal.eu (Jon Ribbens)
Date: Sat, 5 Nov 2016 09:45:12 -0000 (UTC)
Subject: pip3 : command not found
References: <CAFpV4RpSbFLcBG4=Ax9jXzKEzD1Dg9N+bkST3VG-3NUtHS9Y4g@mail.gmail.com>
 <85zill9wv5.fsf@benfinney.id.au>
 <mailman.13.1477874313.17234.python-list@python.org>
 <slrno1dvok.2d7.jon+usenet@sable.unequivocal.eu>
 <5817231d$0$1593$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <slrno1rai3.sd6.jon+usenet@sable.unequivocal.eu>

On 2016-10-31, Steve D'Aprano <steve+python at pearwood.info> wrote:
> On Mon, 31 Oct 2016 07:21 pm, Jon Ribbens wrote:
>> On 2016-10-31, Ben Finney <ben+python at benfinney.id.au> wrote:
>>> Instead, you should invoke the exact Python interpreter you want ? and,
>>> by extension, the Python environment into which you want packages
>>> installed.
>>>
>>>     $ /foo/bar/virtualenv/bin/python3 -m pip install LoremIpsum
>> 
>> I'm slightly curious about that. /foo/bar/virtualenv/bin/python3
>> will just be a symbolic link to /usr/bin/python3, so how does
>> invoking the intepreter that way make any difference?
>
> It doesn't. If you read the rest of Ben's post, or for that matter the
> subject line of this thread, you will see he is comparing:
>
>     path/to/python3 -m pip install LoremIpsum
>
> against:
>
>     pip3 install LoremIpsum

No, if you read the rest of Ben's post you will see that that is not
what he wrote. Maybe he meant what you are saying, I don't know, but
it isn't what he wrote. He clearly implied that you can run Python
in the context of a virtualenv by just invoking that virtualenv's
local Python without running 'activate' first. I'm curious as to
whether this is true or not (how virtualenvs work seems to be very
opaque).


From arthurhavlicek at gmail.com  Sat Nov  5 06:50:11 2016
From: arthurhavlicek at gmail.com (Arthur Havlicek)
Date: Sat, 5 Nov 2016 11:50:11 +0100
Subject: Pre-pep discussion material: in-place equivalents to map and
 filter
In-Reply-To: <581d9b90$0$1599$c3e8da3$5496439d@news.astraweb.com>
References: <8a268e42-87c4-4054-bd19-1dc417dd6d01@googlegroups.com>
 <581af57a$0$1585$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmrQdo2J2ztRz-waim2rsaugi-CJDzOrFe441-OqgoS9Pg@mail.gmail.com>
 <mailman.17.1478181910.7975.python-list@python.org>
 <581b6d3a$0$1583$c3e8da3$5496439d@news.astraweb.com>
 <CAPpswAJF7PzVb1nyqD0fj=2SuW7pT6QNrH_vz1+CVSNGU-Pv2Q@mail.gmail.com>
 <mailman.26.1478204949.7975.python-list@python.org>
 <581d9b90$0$1599$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPpswAJQQQAr2eZayvARMZf--BFM_D7PWoitzbmvvqfAQgzB0w@mail.gmail.com>

2016-11-05 9:42 GMT+01:00 Steve D'Aprano <steve+python at pearwood.info>:

>
> I don't know who you are quoting there. It is considered impolite to quote
> people without giving attribution, and makes it harder to respond.
>

My bad. I was unaware of that. This was quoted from Ned Batchelder's mali.


2016-11-05 9:42 GMT+01:00 Steve D'Aprano <steve+python at pearwood.info>:

> I trust that isn't actually the case, but by describing decorators and
> lambda as too fancy to use, you're giving a good impression of somebody who
> doesn't know what they're doing.
>

I was a bit humorous here, making fun about the fact I do not write
anything sophisticated.

However, the fact that we do avoid them is entirely true ! We do so because
these concepts are foreign to the average developer (lambda being a bit
apart: it's easily understood, but functional constructs may not, so its a
result of us avoiding functional constructs as well).

Pick 10 programmers for hire and count how many know how to write a
decorator. If you have specified you needed python specialists, you may
have 3-4. If not, you are lucky to find even one. And where I live we don't
have the luxury of finding competent Python experts by kicking a tree. In
our open space, we are 6 devs (+ one ex-dev being my manager), only 2 had
previous Python experience: me and the CEO. And the other 4 are doing fine
! Locking ourselves in Python-specific semantics, over time, will make us
loose precious dev time. The best code is the one everyone can understand.
That is why I point this as "complex, non-obvious", and my manager wouldn't
have so kind words (but he is a Java fan, so he doesn't count.)


2016-11-05 9:42 GMT+01:00 Steve D'Aprano <steve+python at pearwood.info>:

> - you would have to demonstrate a good reason to think that this new
> feature
> will actually be more efficient and faster
>

This is easy but time-consuming, I could roll my implementation and
showcase a few benchs. I am very confident that is the case. I would have
bet that I would need to do it at some point, but I wanted to poll opinions
a bit beforehand.


2016-11-05 9:42 GMT+01:00 Steve D'Aprano <steve+python at pearwood.info>:

> In other words, in my opinion:
>
> - you would need to prove that there is a widespread need for in-place
> modification of lists, where the lists are big enough that using a
> temporary list is not practical
>

However this is tricky, I may very well be an exception.

About your bench: I don't really know why you are surprised the for loop is
slower. That's the result of comprehension being native while for loop
isn't. That does not mean writing to a copy would save time for exchange of
memory. In my opinion, the fact that we will win something is guaranteed
because we save a copy call and do the exact same operation. There is
nothing like cache magic optimization that could happen because the mapping
needs to read the first list anyway. Nor we need a temporary buffer to
cache operations since they are independent. Really, I am ready to make a
serious bet.


From steve+python at pearwood.info  Sat Nov  5 07:23:24 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sat, 05 Nov 2016 22:23:24 +1100
Subject: pip3 : command not found
References: <CAFpV4RpSbFLcBG4=Ax9jXzKEzD1Dg9N+bkST3VG-3NUtHS9Y4g@mail.gmail.com>
 <85zill9wv5.fsf@benfinney.id.au>
 <mailman.13.1477874313.17234.python-list@python.org>
 <slrno1dvok.2d7.jon+usenet@sable.unequivocal.eu>
 <5817231d$0$1593$c3e8da3$5496439d@news.astraweb.com>
 <slrno1rai3.sd6.jon+usenet@sable.unequivocal.eu>
Message-ID: <581dc12e$0$1617$c3e8da3$5496439d@news.astraweb.com>

On Sat, 5 Nov 2016 08:45 pm, Jon Ribbens wrote:

> On 2016-10-31, Steve D'Aprano <steve+python at pearwood.info> wrote:
>> On Mon, 31 Oct 2016 07:21 pm, Jon Ribbens wrote:
>>> On 2016-10-31, Ben Finney <ben+python at benfinney.id.au> wrote:
>>>> Instead, you should invoke the exact Python interpreter you want ? and,
>>>> by extension, the Python environment into which you want packages
>>>> installed.
>>>>
>>>>     $ /foo/bar/virtualenv/bin/python3 -m pip install LoremIpsum
>>> 
>>> I'm slightly curious about that. /foo/bar/virtualenv/bin/python3
>>> will just be a symbolic link to /usr/bin/python3, so how does
>>> invoking the intepreter that way make any difference?
>>
>> It doesn't. If you read the rest of Ben's post, or for that matter the
>> subject line of this thread, you will see he is comparing:
>>
>>     path/to/python3 -m pip install LoremIpsum
>>
>> against:
>>
>>     pip3 install LoremIpsum
> 
> No, if you read the rest of Ben's post you will see that that is not
> what he wrote. 

Excuse me, that is what he wrote. As you yourself quoted.

The OP said he tried to call the "pip3" command, and got an error that the
command was not found. Ben replied:

"There is no guarantee that a command named ?pip3? will be installed."

and then stated:

"Instead, you should invoke the exact Python interpreter you want"

The fact that (and now I'm quoting *you*, not Ben) 

    /foo/bar/virtualenv/bin/python3 will just be a symbolic link
    to /usr/bin/python3

is irrelevant. It doesn't matter whether you call python3 via the actual
executable file, or the symbolic link. But Ben never suggested that it
would make a difference: he was contrasting calling the actual Python
interpreter wanted (Python 3, via a virtualenv) with calling a command pip3
(which does not actually exist on the OP's system).


Your implied question here:

> Maybe he meant what you are saying, I don't know, but 
> it isn't what he wrote. He clearly implied that you can run Python
> in the context of a virtualenv by just invoking that virtualenv's
> local Python without running 'activate' first. I'm curious as to
> whether this is true or not (how virtualenvs work seems to be very
> opaque).

is a separate issue from the symbolic link question. Possibly you do have to
run `activate` first, I don't know, but either way this was not part of
your earlier question. You said nothing about `activate` in your earlier
post, and this is the first time you have mentioned it.

The bottom line is, I couldn't have answered your question about `activate`
because you hadn't asked it yet; you are correct that there's no difference
between calling Python via the executable and via a symlink; but Ben never
said that there was such a difference.


This thread reminds me of a scene from the Marx Bros movie "At The Circus".
Alas, Google has let me down as far as the exact quote, but it goes
something like this:

Groucho Marx, accidentally letting the big secret slip: 
    "The elephants will be here soon."

Society dame Margaret Dumont, taken aback:
    "Elephants? What elephants?"

Groucho:
    "Nobody said anything about elephants!"

    [shakes head disgustedly]

    "Elephants! At your age!"


I'll let you decide whether I'm Groucho or Dumont.



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From rosuav at gmail.com  Sat Nov  5 07:47:16 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 5 Nov 2016 22:47:16 +1100
Subject: Pre-pep discussion material: in-place equivalents to map and
 filter
In-Reply-To: <CAPpswAJQQQAr2eZayvARMZf--BFM_D7PWoitzbmvvqfAQgzB0w@mail.gmail.com>
References: <8a268e42-87c4-4054-bd19-1dc417dd6d01@googlegroups.com>
 <581af57a$0$1585$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmrQdo2J2ztRz-waim2rsaugi-CJDzOrFe441-OqgoS9Pg@mail.gmail.com>
 <mailman.17.1478181910.7975.python-list@python.org>
 <581b6d3a$0$1583$c3e8da3$5496439d@news.astraweb.com>
 <CAPpswAJF7PzVb1nyqD0fj=2SuW7pT6QNrH_vz1+CVSNGU-Pv2Q@mail.gmail.com>
 <mailman.26.1478204949.7975.python-list@python.org>
 <581d9b90$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <CAPpswAJQQQAr2eZayvARMZf--BFM_D7PWoitzbmvvqfAQgzB0w@mail.gmail.com>
Message-ID: <CAPTjJmoyg=QgUbjUvb8EsvL4sS35NAp8VXOfQ1xNxyE_wNqBgQ@mail.gmail.com>

On Sat, Nov 5, 2016 at 9:50 PM, Arthur Havlicek
<arthurhavlicek at gmail.com> wrote:
> Pick 10 programmers for hire and count how many know how to write a
> decorator. If you have specified you needed python specialists, you may
> have 3-4. If not, you are lucky to find even one.

By "write a decorator", I presume you mean implement the decorator
function, because anyone who's used Flask will have used "@app.route",
for instance.

But here's the thing. For everyone who writes a decorator function,
there could be dozens who use it. How many people *on this mailing
list* know how to implement namedtuple, vs how how many know how to
use it? How many know how to properly implement a cache, vs how many
can slap "@lru_cache()" on top of a function? For the most utterly
extreme example possible, how many people in the world know how to
encrypt data securely, vs how many know how to put "https:" into their
browsers? When you look at "programmers for hire", you get a small
number of brilliant experts with huge amounts of experience, and the
rest are split between mid-level people looking for a new job, and
entry-level people straight out of college. (Depending on your
environment, the stats could easily be skewed either direction.) Most
entry-level programmers are not going to have experience with writing
decorator functions - but they don't need it. (Not that it's that
hard. They're functions that accept and return functions, that's all.)

> This is easy but time-consuming, I could roll my implementation and
> showcase a few benchs. I am very confident that is the case. I would have
> bet that I would need to do it at some point, but I wanted to poll opinions
> a bit beforehand.
>
> About your bench: I don't really know why you are surprised the for loop is
> slower. That's the result of comprehension being native while for loop
> isn't. That does not mean writing to a copy would save time for exchange of
> memory. In my opinion, the fact that we will win something is guaranteed
> because we save a copy call and do the exact same operation. There is
> nothing like cache magic optimization that could happen because the mapping
> needs to read the first list anyway. Nor we need a temporary buffer to
> cache operations since they are independent. Really, I am ready to make a
> serious bet.

Okay! *Do* make that bet. Gamble the most precious currency there is:
developer time. How much are you prepared to bet? Two hours? Ten
hours? Spend that much time writing the implementation and
benchmarking it. If you're confident that this really is such a win,
the time won't be lost - you'll end up with a viable patch for
inclusion. If you lose the bet, well, that's what betting is like.
Sometimes you lose.

I may sound cynical and critical from the above ("show me"), but in
reality, I would love to see the results of your benchmark. Improving
performance is a good thing. I just want to know that it's actually
going to happen.

ChrisA


From rosuav at gmail.com  Sat Nov  5 08:05:19 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 5 Nov 2016 23:05:19 +1100
Subject: Django Application Model Reference
In-Reply-To: <e1e80737-4a33-4e5d-b53f-1188c0cc9225@googlegroups.com>
References: <c6382323-fdbe-447d-9229-08c9b61e1ed6@googlegroups.com>
 <e1e80737-4a33-4e5d-b53f-1188c0cc9225@googlegroups.com>
Message-ID: <CAPTjJmriq8KFBT3SMF4WSuLAZSNoKKD_Pz+MyKAtybaAh8RNaQ@mail.gmail.com>

On Thu, Nov 3, 2016 at 4:44 AM, Dreyton Scott <drekscott at gmail.com> wrote:
> On Wednesday, November 2, 2016 at 1:40:35 PM UTC-4, Dreyton Scott wrote:
>> Hello. I am currently creating a notification django application that will need to be able to "hook" into another django application. What my app needs is a way to retrieve all model classes in the connected application. Is there a way to do this?
>
> In a way such that my app (a) is connected to app (b). Have (a) retrieve all model class instances in the connected app (b). Is that possible?

You may have to elaborate on "connected" here. Is it able to import
the code of the other app, or is it restricted to HTTP requests and
responses? If the latter, there's no significance to them both being
written in Django; you'll have to create an explicit endpoint that
enumerates the models, probably sending the info as a JSON object.
(Caveat: I don't know Django, but I do know HTTP.)

ChrisA


From neha.agrawal1428 at gmail.com  Sat Nov  5 08:35:29 2016
From: neha.agrawal1428 at gmail.com (neha.agrawal1428 at gmail.com)
Date: Sat, 5 Nov 2016 05:35:29 -0700 (PDT)
Subject: mentor training python Romania with certification
In-Reply-To: <d1bc2b9b-923f-4e81-9407-8516c60fb2b1@googlegroups.com>
References: <f7651e03-45d8-49a9-af96-6dece61dba0a@googlegroups.com>
 <d1bc2b9b-923f-4e81-9407-8516c60fb2b1@googlegroups.com>
Message-ID: <7a3c695c-7d20-4227-8e95-e5e3ebce359d@googlegroups.com>

On Monday, September 5, 2016 at 3:33:55 PM UTC+5:30, neha.agr... at gmail.com wrote:
> On Tuesday, August 16, 2016 at 6:45:04 PM UTC+5:30, blue wrote:
> > Hi. 
> > 
> > I'm from Romania.
> > I need to update my skils under python.
> > I need to find one mentor ( class, training ) to obtain one certified under python language.
> > 
> > C?t?lin George Fe?til?
> 
> Hi,
> 
> You might find Python course offered by edureka to be useful
> http://www.edureka.co/python
> 
> Check it out!
> 
> Regards
> Neha

Hi,

Please check out the below link for a demo on Edureka's Python course
http://unbouncepages.com/python-training-3/

Regards
Neha


From salihuabdu at gmail.com  Sat Nov  5 09:33:13 2016
From: salihuabdu at gmail.com (Abdullahi Salihu Abubakar)
Date: Sat, 5 Nov 2016 14:33:13 +0100
Subject: No subject
Message-ID: <CA++xw78t1YmN=_WKHEX7u_GUPMOdUz6enWhR1VDh=aNG5fYvOw@mail.gmail.com>

Hello!


Hope you guys are fine.  I am a newbie in programming, though I have been
experimenting with Python for close to 3 years, but have never attempt
writing any real-life program.  However, recently I decided to embark on a
personal project to develop a Python App which I want to dedicate to my
community to solve a particular calculation challenge relating to financial
benefits involving a combination of beneficiaries with different level of
benefits.  My intention is to use Python 3.4 or 2.7, with Tkinter and
Python Megawidgets (Pmw) for the GUI.  The following is a short description
of the workings of the proposed app:


1.   The user inputs the amount to be distributed to the beneficiaries into
a text entry box;

2.   He clicks add button, to add the value;

3.   He selects the beneficiaries from a dropdown list, one at a time;

4.   He clicks an ?add beneficiary? button to add to the list;

5.   He then clicks a button to calculate the amount entitled by each
beneficiary;

6.   The application then calculates and displays the result in the
following format:

a.   Name: Identity of the beneficiary;

b.   Percentage: the percentage he is entitled from the amount entered (eg.
4/24, 3/12 etc);

c.   Value: the monetary value he is entitled (eg $50 for instance)

7.   The user then clicks a close button to exit the result window.



I don?t have much challenge setting the GUI using Tkinter and Pmw.  But my
challenge is how to handle the data to be entered, calculate the value
based on the percentage each beneficiary is entitled, and allocates same to
him/her.  This is because:

1.   The user can select 3 or 4 beneficiaries with different percentages;

2.   Some beneficiaries don?t have a specific percentage, but only got
their benefits from a reminder value;

3.   Some beneficiaries can have a specific percentage and equally be
entitled to a section of the reminder.

4.   There may be many scenarios (can extend to 1500, but I want to start
experimenting with just 20 instances);

I am not too depth in Python but it is the language of my choice since
2010. I don?t know the complex methods which Python provides in order to
solve this intricate problem for me.  I attempted using the following
python codes:


<code>


# I create different categories of beneficiaries first, using a list.


cat1 = [element1, element2, element3, element4]



# Capture the values from the user


amount = input(?Enter Value: ?)


choice = input(?Enter Beneficiary:  ?)



# I define a defaultdict object in order to set the values for each
beneficiary, as it allows for one-to-many relationship.


Result = defaultdict(list)


# I now use control flow statements to compare values and distribute.


if choice == cat1:

        result[cat1[0]].append((amount/6) * 1)

        result[cat1[1]].append((amount/6) * 1)

        result[cat1[2]].append((amount/6) * 2)

        result[cat1[3]].append((amount/3) * 2)

elif choice == cat2:

        ?continue same approach as above.



</code>


My Dilemma


My challenge here is; the above approach does not provide me the necessary
solution. Because:

1.   Let?s assume the first part of the if statement gave me the necessary
values for the beneficiaries, but running it does not produce anything. It
only outputs the storage location of the ?result? defaultdict object.  What
is the best approach, please?

2.   I got stocked because I can?t continue using the ?if statement? for
each and every category of beneficiaries in order to check the user input,
since I have to create as many instance as possible;

3.   I can?t figure out how to handle the result.  Which method can I use
to display the result based on the format given above?

4.   I presently use list to hold the list of beneficiaries. Based on the
short description of the proposed app above, which other container is it
suitable for holding the categories of the beneficiaries?

I am sorry if I omit any necessary information. This is how far I have
gone. I am always ready to provide any necessary details please. Am a
newbie, but always ready to learn, no matter what.


Thanks in anticipation of your magnanimity.



-- 
Abdallah Salihu Abubakar (Abu Siddeeq)
Off Lagos crescent,
Unguwar Hausawa,
Garki Village,
P. O. Box 11617,
Garki 900001,
Abuja - Nigeria
08034592444
salihuabdu at gmail.com, http://groups.yahoo.com/groups/nurul-islam


From salihuabdu at gmail.com  Sat Nov  5 09:37:02 2016
From: salihuabdu at gmail.com (Abdullahi Salihu Abubakar)
Date: Sat, 5 Nov 2016 06:37:02 -0700 (PDT)
Subject: A Python App To Calculate Sum Of Different Elements In A List With
 Different Values
Message-ID: <720898c1-ac47-4cb4-8a4a-c4ef9f22d7d2@googlegroups.com>

Hello!

Hope you guys are fine.  I am a newbie in programming, though I have been experimenting with Python for close to 3 years, but have never attempt writing any real-life program.  However, recently I decided to embark on a personal project to develop a Python App which I want to dedicate to my community to solve a particular calculation challenge relating to financial benefits involving a combination of beneficiaries with different level of benefits.  My intention is to use Python 3.4 or 2.7, with Tkinter and Python Megawidgets (Pmw) for the GUI.  The following is a short description of the workings of the proposed app:

1.   The user inputs the amount to be distributed to the beneficiaries into a text entry box;

2.   He clicks add button, to add the value;

3.   He selects the beneficiaries from a dropdown list, one at a time;

4.   He clicks an ?add beneficiary? button to add to the list;

5.   He then clicks a button to calculate the amount entitled by each beneficiary;

6.   The application then calculates and displays the result in the following format:

a.   Name: Identity of the beneficiary;

b.   Percentage: the percentage he is entitled from the amount entered (eg. 4/24, 3/12 etc);

c.   Value: the monetary value he is entitled (eg $50 for instance)

7.   The user then clicks a close button to exit the result window.

 
I don?t have much challenge setting the GUI using Tkinter and Pmw.  But my challenge is how to handle the data to be entered, calculate the value based on the percentage each beneficiary is entitled, and allocates same to him/her.  This is because:
1.   The user can select 3 or 4 beneficiaries with different percentages;

2.   Some beneficiaries don?t have a specific percentage, but only got their benefits from a reminder value;

3.   Some beneficiaries can have a specific percentage and equally be entitled to a section of the reminder.

4.   There may be many scenarios (can extend to 1500, but I want to start experimenting with just 20 instances);

I am not too depth in Python but it is the language of my choice since 2010. I don?t know the complex methods which Python provides in order to solve this intricate problem for me.  I attempted using the following python codes:

<code>

# I create different categories of beneficiaries first, using a list.

cat1 = [element1, element2, element3, element4]
 
# Capture the values from the user

amount = input(?Enter Value: ?)

choice = input(?Enter Beneficiary:  ?)
 
# I define a defaultdict object in order to set the values for each beneficiary, as it allows for one-to-many relationship.

Result = defaultdict(list)

# I now use control flow statements to compare values and distribute.

if choice == cat1:
        result[cat1[0]].append((amount/6) * 1)
        result[cat1[1]].append((amount/6) * 1)
        result[cat1[2]].append((amount/6) * 2)
        result[cat1[3]].append((amount/3) * 2)
elif choice == cat2:
        ?continue same approach as above.
 
</code>

My Dilemma

My challenge here is; the above approach does not provide me the necessary solution. Because:
1.   Let?s assume the first part of the if statement gave me the necessary values for the beneficiaries, but running it does not produce anything. It only outputs the storage location of the ?result? defaultdict object.  What is the best approach, please?

2.   I got stocked because I can?t continue using the ?if statement? for each and every category of beneficiaries in order to check the user input, since I have to create as many instance as possible;

3.   I can?t figure out how to handle the result.  Which method can I use to display the result based on the format given above?

4.   I presently use list to hold the list of beneficiaries. Based on the short description of the proposed app above, which other container is it suitable for holding the categories of the beneficiaries?  

I am sorry if I omit any necessary information. This is how far I have gone. I am always ready to provide any necessary details please. Am a newbie, but always ready to learn, no matter what.

Thanks in anticipation of your magnanimity.


From salihuabdu at gmail.com  Sat Nov  5 09:39:11 2016
From: salihuabdu at gmail.com (Abdullahi Salihu Abubakar)
Date: Sat, 5 Nov 2016 14:39:11 +0100
Subject: A Python App To Calculate Sum Of Different Elements In A List With
 Different Values
Message-ID: <CA++xw7_du1SDOkqgKf9Qxv6wJD08gvBLWSeS9t1f22PGFdN6fg@mail.gmail.com>

Hello!


Hope you guys are fine.  I am a newbie in programming, though I have been
experimenting with Python for close to 3 years, but have never attempt
writing any real-life program.  However, recently I decided to embark on a
personal project to develop a Python App which I want to dedicate to my
community to solve a particular calculation challenge relating to financial
benefits involving a combination of beneficiaries with different level of
benefits.  My intention is to use Python 3.4 or 2.7, with Tkinter and
Python Megawidgets (Pmw) for the GUI.  The following is a short description
of the workings of the proposed app:


1.   The user inputs the amount to be distributed to the beneficiaries into
a text entry box;

2.   He clicks add button, to add the value;

3.   He selects the beneficiaries from a dropdown list, one at a time;

4.   He clicks an ?add beneficiary? button to add to the list;

5.   He then clicks a button to calculate the amount entitled by each
beneficiary;

6.   The application then calculates and displays the result in the
following format:

a.   Name: Identity of the beneficiary;

b.   Percentage: the percentage he is entitled from the amount entered (eg.
4/24, 3/12 etc);

c.   Value: the monetary value he is entitled (eg $50 for instance)

7.   The user then clicks a close button to exit the result window.



I don?t have much challenge setting the GUI using Tkinter and Pmw.  But my
challenge is how to handle the data to be entered, calculate the value
based on the percentage each beneficiary is entitled, and allocates same to
him/her.  This is because:

1.   The user can select 3 or 4 beneficiaries with different percentages;

2.   Some beneficiaries don?t have a specific percentage, but only got
their benefits from a reminder value;

3.   Some beneficiaries can have a specific percentage and equally be
entitled to a section of the reminder.

4.   There may be many scenarios (can extend to 1500, but I want to start
experimenting with just 20 instances);

I am not too depth in Python but it is the language of my choice since
2010. I don?t know the complex methods which Python provides in order to
solve this intricate problem for me.  I attempted using the following
python codes:


<code>


# I create different categories of beneficiaries first, using a list.


cat1 = [element1, element2, element3, element4]



# Capture the values from the user


amount = input(?Enter Value: ?)


choice = input(?Enter Beneficiary:  ?)



# I define a defaultdict object in order to set the values for each
beneficiary, as it allows for one-to-many relationship.


Result = defaultdict(list)


# I now use control flow statements to compare values and distribute.


if choice == cat1:

        result[cat1[0]].append((amount/6) * 1)

        result[cat1[1]].append((amount/6) * 1)

        result[cat1[2]].append((amount/6) * 2)

        result[cat1[3]].append((amount/3) * 2)

elif choice == cat2:

        ?continue same approach as above.



</code>


My Dilemma


My challenge here is; the above approach does not provide me the necessary
solution. Because:

1.   Let?s assume the first part of the if statement gave me the necessary
values for the beneficiaries, but running it does not produce anything. It
only outputs the storage location of the ?result? defaultdict object.  What
is the best approach, please?

2.   I got stocked because I can?t continue using the ?if statement? for
each and every category of beneficiaries in order to check the user input,
since I have to create as many instance as possible;

3.   I can?t figure out how to handle the result.  Which method can I use
to display the result based on the format given above?

4.   I presently use list to hold the list of beneficiaries. Based on the
short description of the proposed app above, which other container is it
suitable for holding the categories of the beneficiaries?

I am sorry if I omit any necessary information. This is how far I have
gone. I am always ready to provide any necessary details please. Am a
newbie, but always ready to learn, no matter what.


Thanks in anticipation of your magnanimity.

--


From arthurhavlicek at gmail.com  Sat Nov  5 09:50:31 2016
From: arthurhavlicek at gmail.com (Arthur Havlicek)
Date: Sat, 5 Nov 2016 14:50:31 +0100
Subject: Pre-pep discussion material: in-place equivalents to map and
 filter
In-Reply-To: <CAPTjJmoyg=QgUbjUvb8EsvL4sS35NAp8VXOfQ1xNxyE_wNqBgQ@mail.gmail.com>
References: <8a268e42-87c4-4054-bd19-1dc417dd6d01@googlegroups.com>
 <581af57a$0$1585$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmrQdo2J2ztRz-waim2rsaugi-CJDzOrFe441-OqgoS9Pg@mail.gmail.com>
 <mailman.17.1478181910.7975.python-list@python.org>
 <581b6d3a$0$1583$c3e8da3$5496439d@news.astraweb.com>
 <CAPpswAJF7PzVb1nyqD0fj=2SuW7pT6QNrH_vz1+CVSNGU-Pv2Q@mail.gmail.com>
 <mailman.26.1478204949.7975.python-list@python.org>
 <581d9b90$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <CAPpswAJQQQAr2eZayvARMZf--BFM_D7PWoitzbmvvqfAQgzB0w@mail.gmail.com>
 <CAPTjJmoyg=QgUbjUvb8EsvL4sS35NAp8VXOfQ1xNxyE_wNqBgQ@mail.gmail.com>
Message-ID: <CAPpswAJebKkALVmmzDd45fOU_G34QGUAJfuZ8exeDPcZSGfC3g@mail.gmail.com>

2016-11-05 12:47 GMT+01:00 Chris Angelico <rosuav at gmail.com>:

> On Sat, Nov 5, 2016 at 9:50 PM, Arthur Havlicek
>
> But here's the thing. For everyone who writes a decorator function,
> there could be dozens who use it.
>

The day that one guy leaves the team, suddenly you have code that's become
a bit tricky to maintain. Our philosophy is that everyone should be able to
contribute anywhere although I agree with you: there is such thing as using
a decorator as a black box, and even diving into it isn't that hard. Notice
I didn't say we don't use these features at all, we just tend to avoid them
when a more familiar approach exist. Even so, maybe we are wrong to do so,
and the code would be clearer if we used more features. Maybe we just lack
the expertise and we are indeed stuck in an old mindset, after all. Anyway,
I have some directives to make my code easy to access and must compose with
what I would call industrial constraints.

As a gambler I can't refuse that bet proposal, no sir :) Expect me to come
back with a piece of code and/or questions about CPython engine.


From jon+usenet at unequivocal.eu  Sat Nov  5 11:55:54 2016
From: jon+usenet at unequivocal.eu (Jon Ribbens)
Date: Sat, 5 Nov 2016 15:55:54 -0000 (UTC)
Subject: pip3 : command not found
References: <CAFpV4RpSbFLcBG4=Ax9jXzKEzD1Dg9N+bkST3VG-3NUtHS9Y4g@mail.gmail.com>
 <85zill9wv5.fsf@benfinney.id.au>
 <mailman.13.1477874313.17234.python-list@python.org>
 <slrno1dvok.2d7.jon+usenet@sable.unequivocal.eu>
 <5817231d$0$1593$c3e8da3$5496439d@news.astraweb.com>
 <slrno1rai3.sd6.jon+usenet@sable.unequivocal.eu>
 <581dc12e$0$1617$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <slrno1s095.sd6.jon+usenet@sable.unequivocal.eu>

On 2016-11-05, Steve D'Aprano <steve+python at pearwood.info> wrote:
> Your implied question here:
>
>> Maybe he meant what you are saying, I don't know, but 
>> it isn't what he wrote. He clearly implied that you can run Python
>> in the context of a virtualenv by just invoking that virtualenv's
>> local Python without running 'activate' first. I'm curious as to
>> whether this is true or not (how virtualenvs work seems to be very
>> opaque).
>
> is a separate issue from the symbolic link question. Possibly you do have to
> run `activate` first, I don't know, but either way this was not part of
> your earlier question. You said nothing about `activate` in your earlier
> post, and this is the first time you have mentioned it.

I'm afraid I can only suggest that you try re-reading the subthread
again until you manage to understand it. It wasn't really that
complicated but you seem to have confused yourself greatly.


From steve+python at pearwood.info  Sat Nov  5 12:53:20 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sun, 06 Nov 2016 03:53:20 +1100
Subject: pip3 : command not found
References: <CAFpV4RpSbFLcBG4=Ax9jXzKEzD1Dg9N+bkST3VG-3NUtHS9Y4g@mail.gmail.com>
 <85zill9wv5.fsf@benfinney.id.au>
 <mailman.13.1477874313.17234.python-list@python.org>
 <slrno1dvok.2d7.jon+usenet@sable.unequivocal.eu>
 <5817231d$0$1593$c3e8da3$5496439d@news.astraweb.com>
 <slrno1rai3.sd6.jon+usenet@sable.unequivocal.eu>
 <581dc12e$0$1617$c3e8da3$5496439d@news.astraweb.com>
 <slrno1s095.sd6.jon+usenet@sable.unequivocal.eu>
Message-ID: <581e0e81$0$1598$c3e8da3$5496439d@news.astraweb.com>

On Sun, 6 Nov 2016 02:55 am, Jon Ribbens wrote:

> I'm afraid I can only suggest that you try re-reading the subthread
> again until you manage to understand it. It wasn't really that
> complicated but you seem to have confused yourself greatly.

Are you serious?

Okay. Here's the start of the thread:

https://mail.python.org/pipermail/python-list/2016-October/715993.html

where the OP asks for help because running "pip3" causes a "command not
found" error. Just as the subject line says.

Here's Ben's reply:

https://mail.python.org/pipermail/python-list/2016-October/715994.html

where he clearly says not to rely on the "pip3" command, as it may not
exist, or may not be in the environment you expect, but to specify
precisely which Python interpreter you want to run:

    [quote]
    Instead, you should invoke the exact Python interpreter you want


And here's your response:

https://mail.python.org/pipermail/python-list/2016-October/716005.html

where you quote those exact same words from Ben and ask what difference it
makes as it is a symbolic link. You'll note that there is not one word
about "activate" in your post, just as I said. Instead you question why Ben
thinks there's a difference between running the sym link and running the
direct link to the executable, even though that's not what Ben said.

I can only repeat yet again: you have not understood Ben. He's not stating
that there is a difference between:

    /direct/path/to/python3.x

and 

    /symbolic/link/to/direct/path/to/python3.x


You are right to state that they will be exactly the same[1], but wrong to
imagine that Ben said that they were different. That's not what Ben said.
He stated that there is a difference between:

    /direct/path/to/python3.x -m pip ...

which specifies the precise Python environment you want to run, and:

    pip3 ... 

which may not exist at all, or if it exists it may not be on the PATH, or if
it is on the PATH it may not install into the Python environment that you
expect.


Oh, and ironically, you'll see that Ben anticipated and answered your
question about activate. Here's the link again, to save you scrolling up:

https://mail.python.org/pipermail/python-list/2016-October/715994.html

where he says:

    If you already have a specific environment active and know that
    ?python3? is the correct Python interpreter from that environment,
    you can omit the explicit path.


The implication is that the answer to your question is Yes, you can run
Python in the context of a virtualenv by just invoking that virtualenv's
local Python without running 'activate' first. Is Ben correct? The
virtualenv documentation confirms that Ben is right:

    If you directly run a script or the python interpreter from the
    virtualenv?s bin/ directory (e.g. path/to/ENV/bin/pip or 
    /path/to/ENV/bin/python-script.py) there?s no need for activation.

https://virtualenv.pypa.io/en/stable/userguide/#activate-script






[1] Technically, the application being run may invoke different behaviour
depending on the name it was invoked by. Some editors do that, e.g.
the "joe" editor. But I don't believe Python does anything like this.

http://joe-editor.sourceforge.net/

-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From mr at e-wrobel.pl  Sat Nov  5 13:10:37 2016
From: mr at e-wrobel.pl (Mr. Wrobel)
Date: Sat, 5 Nov 2016 18:10:37 +0100
Subject: [Theory] How to speed up python code execution / pypy vs GPU
Message-ID: <nvl3q7$vut$1@news.icm.edu.pl>

Hi,

Some skeptics asked my why there is a reason to use Python against of 
any other "not interpreted" languages, like objective-C. As my 
explanation, I have answered that there is a a lot of useful APIs, 
language is modern, has advanced objective architecture, and what is the 
most important - it is  dynamic and support is simply great.

However the same skeptics told my that, ok we believe that it is true, 
however the code execution is much slower than any other compiled language.

I must tell you that is the reason I started to dig into internet and 
searching some methods to speed up python's code.

1. What I have found is modified python interpreter - pypy  - 
http://pypy.org that does not require any different approach to develop 
your code.

2. And: Gpu based computing powered by Nvidia (NumbaPro compiler): 
https://developer.nvidia.com/how-to-cuda-python

Do you have any experience in that areas, and maybe some benchmarks 
showing advantage of using these technologies?

Cheers,
Marcin


From steve+python at pearwood.info  Sat Nov  5 13:12:07 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sun, 06 Nov 2016 04:12:07 +1100
Subject: First security bug related to f-strings
Message-ID: <581e12e9$0$1601$c3e8da3$5496439d@news.astraweb.com>

Well, that didn't take very long at all.

Here's the first security bug which is related to the new (and badly
misnamed) f-string feature:

http://bugs.python.org/issue28563

Note what I'm not saying: I'm not saying that the bug is *caused* by
f-strings. It is not. The bug is actually caused by the inappropriate use
of eval. But the worrying thing here is:


    Bonus: With the new string interpolation in Python 3.7, exploiting 
    gettext.c2py becomes trivial:

       gettext.c2py('f"{os.system(\'sh\')}"')(0)

    The tokenizer will recognize the entire format-string as just a string,
    thus bypassing the security checks.


Yay for hiding arbitrary code evaluation inside something that pretends to
be a string!

My guess is that there is probably more code out there in the wild which is
vulnerable to code injection attacks thanks to the use of eval or exec, but
its been too hard to exploit up until now. But with f-strings, chances are
that they too will be trivially exploitable.



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From bill at baddogconsulting.com  Sat Nov  5 13:30:52 2016
From: bill at baddogconsulting.com (Bill Deegan)
Date: Sat, 5 Nov 2016 13:30:52 -0400
Subject: SCons 2.5.1 Released
Message-ID: <CAEyG4CGJ2guh5RKpM2EXfEaUue0ej5_qCGreoM-fsZ91w7hMsA@mail.gmail.com>

Available at:

https://sourceforge.net/projects/scons/files/latest/download?source=files

Changelog:

                 SCons - a software construction tool

                            Change Log

RELEASE 2.5.1 - Mon, 03 Nov 2016 13:37:42 -0400

  From William Deegan:
    - Add scons-configure-cache.py to packaging. It was omitted

  From Alexey Klimkin:
    - Use memoization to optimize PATH evaluation across all dependencies
per
      node. (PR #345)


RELEASE 2.5.0 - Mon, 09 Apr 2016 11:27:42 -0700

  From Dirk Baechle:
    - Removed a lot of compatibility methods and workarounds
      for Python versions < 2.7, in order to prepare the work
      towards a combined 2.7/3.x version. (PR #284)
      Also fixed the default arguments for the print_tree and
      render_tree methods. (PR #284, too)

  From William Blevins:
    - Added support for cross-language dependency scanning;
      SCons now respects scanner keys for implicit dependencies.
      - Notes for SCons users with heterogeneous systems.
        - May find new (previously missed) dependencies.
        - May cause rebuild after upgrade due to dependency changes.
        - May find new dependency errors (EG. cycles).
          - Discovered in some of the SCons QT tests.
    - Resolved missing cross-language dependencies for
      SWIG bindings (fixes #2264).
    - Corrected typo in User Guide for Scanner keyword. (PR #2959)
    - Install builder interacts with scanner found in SCANNERS differently.
      - Previous: Install builder recursively scanned implicit dependencies
        for scanners from SCANNER, but not for built-in (default) scanners.
      - Current: Install builder will not scan for implicit dependencies via
        either scanner source. This optimizes some Install builder behavior
        and brings orthogonality to Install builder scanning behavior.

  From William Deegan:
    - Add better messaging when two environments have
      different actions for the same target (Bug #2024)
    - Fix issue only with MSVC and Always build where targets
      marked AlwaysBuild wouldn't make it into CHANGED_SOURCES
      and thus yield an empty compile command line. (Bug #2622)
    - Fix posix platform escaping logic to properly handle paths
      with parens in them "()".  (Bug #2225)

  From Jakub Pola:
    - Intel Compiler 2016 (Linux/Mac) update for tool directories.

  From Adarsh Sanjeev:
    - Fix for issue #2494: Added string support for Chmod function.

  From Tom Tanner:
    - change cache to use 2 character subdirectories, rather than one
character,
      so as not to give huge directories for large caches, a situation which
      causes issues for NFS.
      For existing caches, you will need to run the scons-configure-cache.py
      script to update them to the new format. You will get a warning every
time
      you build until you co this.
    - Fix a bunch of unit tests on windows


From irmen.NOSPAM at xs4all.nl  Sat Nov  5 13:33:16 2016
From: irmen.NOSPAM at xs4all.nl (Irmen de Jong)
Date: Sat, 5 Nov 2016 18:33:16 +0100
Subject: First security bug related to f-strings
In-Reply-To: <581e12e9$0$1601$c3e8da3$5496439d@news.astraweb.com>
References: <581e12e9$0$1601$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <581e17dd$0$886$e4fe514c@news.xs4all.nl>

On 5-11-2016 18:12, Steve D'Aprano wrote:
> Well, that didn't take very long at all.
> 
> Here's the first security bug which is related to the new (and badly
> misnamed) f-string feature:
> 
> http://bugs.python.org/issue28563


I think perhaps we should have a command line option / environment variable to be able
to disable 'eval' altogether....

Irmen



From eryksun at gmail.com  Sat Nov  5 14:08:44 2016
From: eryksun at gmail.com (eryk sun)
Date: Sat, 5 Nov 2016 18:08:44 +0000
Subject: First security bug related to f-strings
In-Reply-To: <581e17dd$0$886$e4fe514c@news.xs4all.nl>
References: <581e12e9$0$1601$c3e8da3$5496439d@news.astraweb.com>
 <581e17dd$0$886$e4fe514c@news.xs4all.nl>
Message-ID: <CACL+1aukpR=BtA-rbub9Cq81se2eL4cJZ4sD5z-O58Cs23skKA@mail.gmail.com>

On Sat, Nov 5, 2016 at 5:33 PM, Irmen de Jong <irmen.NOSPAM at xs4all.nl> wrote:
> I think perhaps we should have a command line option / environment variable to be able
> to disable 'eval' altogether....

I don't think that's practical. exec and eval are commonly used by
shells and IDEs such as IDLE and IPython. In the standard library,
importlib and namedtuple are two important users of exec. Just try
`import builtins; del builtins.exec, builtins.eval`.


From steve+python at pearwood.info  Sat Nov  5 14:32:18 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sun, 06 Nov 2016 05:32:18 +1100
Subject: [Theory] How to speed up python code execution / pypy vs GPU
References: <nvl3q7$vut$1@news.icm.edu.pl>
Message-ID: <581e25b4$0$1607$c3e8da3$5496439d@news.astraweb.com>

On Sun, 6 Nov 2016 04:10 am, Mr. Wrobel wrote:

> Hi,
> 
> Some skeptics asked my why there is a reason to use Python against of
> any other "not interpreted" languages, like objective-C.

Here's the "Hello World" program in Python:

--- cut ---

print("Hello World")

--- cut ---



Here's the same program in Objective C:

--- cut ---

#import <Foundation/Foundation.h>

int main (int argc, const char * argv[])
{
        NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
        NSLog (@"Hello, World!");
        [pool drain];
        return 0;
}

--- cut ---


Which would you rather write?





-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From irmen.NOSPAM at xs4all.nl  Sat Nov  5 14:50:34 2016
From: irmen.NOSPAM at xs4all.nl (Irmen de Jong)
Date: Sat, 5 Nov 2016 19:50:34 +0100
Subject: First security bug related to f-strings
In-Reply-To: <mailman.44.1478369372.7975.python-list@python.org>
References: <581e12e9$0$1601$c3e8da3$5496439d@news.astraweb.com>
 <581e17dd$0$886$e4fe514c@news.xs4all.nl>
 <CACL+1aukpR=BtA-rbub9Cq81se2eL4cJZ4sD5z-O58Cs23skKA@mail.gmail.com>
 <mailman.44.1478369372.7975.python-list@python.org>
Message-ID: <581e29fc$0$840$e4fe514c@news.xs4all.nl>

On 5-11-2016 19:08, eryk sun wrote:
> On Sat, Nov 5, 2016 at 5:33 PM, Irmen de Jong <irmen.NOSPAM at xs4all.nl> wrote:
>> I think perhaps we should have a command line option / environment variable to be able
>> to disable 'eval' altogether....
> 
> I don't think that's practical. exec and eval are commonly used by
> shells and IDEs such as IDLE and IPython. In the standard library,
> importlib and namedtuple are two important users of exec. Just try
> `import builtins; del builtins.exec, builtins.eval`.
> 

Perhaps. But in those cases you could just leave things on the default.
If you choose to run the interpreter with eval (and exec) disabled, you should be aware
that you'll break tools like that. But for other situations (web server etc) it could
still be useful? I do agree that not being able to use namedtuple (and perhaps other
things from the stdlib) is a problem then.


It was just a thought

Irmen



From rosuav at gmail.com  Sat Nov  5 15:07:54 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 6 Nov 2016 06:07:54 +1100
Subject: pip3 : command not found
In-Reply-To: <581e0e81$0$1598$c3e8da3$5496439d@news.astraweb.com>
References: <CAFpV4RpSbFLcBG4=Ax9jXzKEzD1Dg9N+bkST3VG-3NUtHS9Y4g@mail.gmail.com>
 <85zill9wv5.fsf@benfinney.id.au>
 <mailman.13.1477874313.17234.python-list@python.org>
 <slrno1dvok.2d7.jon+usenet@sable.unequivocal.eu>
 <5817231d$0$1593$c3e8da3$5496439d@news.astraweb.com>
 <slrno1rai3.sd6.jon+usenet@sable.unequivocal.eu>
 <581dc12e$0$1617$c3e8da3$5496439d@news.astraweb.com>
 <slrno1s095.sd6.jon+usenet@sable.unequivocal.eu>
 <581e0e81$0$1598$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPTjJmoLgWY=m8vAJomuujH_wHXGB3uKdSNVUtFFXb11uBhNQQ@mail.gmail.com>

On Sun, Nov 6, 2016 at 3:53 AM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> [1] Technically, the application being run may invoke different behaviour
> depending on the name it was invoked by. Some editors do that, e.g.
> the "joe" editor. But I don't believe Python does anything like this.
>
> http://joe-editor.sourceforge.net/

And quite a few other examples, too - some versions of bash will act
in a more sh-compatible way if invoked as /bin/sh, thus allowing one
to be a link to the other.

Here's how I'm probing Python for this behaviour:

1) Brand new venv

rosuav at sikorsky:~/tmp$ python3 -m venv env

2) With venv active:

(env) rosuav at sikorsky:~/tmp$ python3
Python 3.7.0a0 (default:72e64fc8746b+, Oct 28 2016, 12:35:28)
[GCC 6.2.0 20161010] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys; sys.executable
'/home/rosuav/tmp/env/bin/python3'
>>> sys.path
['', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7',
'/usr/local/lib/python3.7/lib-dynload',
'/home/rosuav/tmp/env/lib/python3.7/site-packages']

3) With venv not active:

rosuav at sikorsky:~$ python3
Python 3.7.0a0 (default:72e64fc8746b+, Oct 28 2016, 12:35:28)
[GCC 6.2.0 20161010] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys; sys.executable
'/usr/local/bin/python3'
>>> sys.path
['', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7',
'/usr/local/lib/python3.7/lib-dynload',
'/home/rosuav/.local/lib/python3.7/site-packages',
'/usr/local/lib/python3.7/site-packages']

4) Proof that it really is a symlink:

(env) rosuav at sikorsky:~/tmp$ ll env/bin
total 32
-rw-r--r-- 1 rosuav rosuav 2134 Nov  6 05:57 activate
-rw-r--r-- 1 rosuav rosuav 1250 Nov  6 05:57 activate.csh
-rw-r--r-- 1 rosuav rosuav 2414 Nov  6 05:57 activate.fish
-rwxr-xr-x 1 rosuav rosuav  249 Nov  6 05:57 easy_install
-rwxr-xr-x 1 rosuav rosuav  249 Nov  6 05:57 easy_install-3.7
-rwxr-xr-x 1 rosuav rosuav  221 Nov  6 05:57 pip
-rwxr-xr-x 1 rosuav rosuav  221 Nov  6 05:57 pip3
-rwxr-xr-x 1 rosuav rosuav  221 Nov  6 05:57 pip3.7
lrwxrwxrwx 1 rosuav rosuav    7 Nov  6 05:57 python -> python3
lrwxrwxrwx 1 rosuav rosuav   22 Nov  6 05:57 python3 -> /usr/local/bin/python3

5) Moment of truth:

rosuav at sikorsky:~$ tmp/env/bin/python3
Python 3.7.0a0 (default:72e64fc8746b+, Oct 28 2016, 12:35:28)
[GCC 6.2.0 20161010] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys; sys.executable
'/home/rosuav/tmp/env/bin/python3'
>>> sys.path
['', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7',
'/usr/local/lib/python3.7/lib-dynload',
'/home/rosuav/tmp/env/lib/python3.7/site-packages']


So it looks like Python actually *does* take notice of the executable
path. This is presumably to allow shebangs to be created the easy way
("#!" + sys.executable) and to always represent the correct Python,
even if you had a venv active.

Now, that said, everything about the previous thread *here* was still
true. People weren't talking about symlinks. But this does mean that
(at least on recent Pythons) running Python from within a venv will
implicitly activate it. (And since the pip* files in that directory
use exactly that shebang, this is also true of running
"tmp/env/bin/pip install spam".)

I'll leave it to someone else to test the older 'virtualenv' module,
as I don't have it installed here.

ChrisA


From eryksun at gmail.com  Sat Nov  5 15:24:29 2016
From: eryksun at gmail.com (eryk sun)
Date: Sat, 5 Nov 2016 19:24:29 +0000
Subject: First security bug related to f-strings
In-Reply-To: <581e29fc$0$840$e4fe514c@news.xs4all.nl>
References: <581e12e9$0$1601$c3e8da3$5496439d@news.astraweb.com>
 <581e17dd$0$886$e4fe514c@news.xs4all.nl>
 <CACL+1aukpR=BtA-rbub9Cq81se2eL4cJZ4sD5z-O58Cs23skKA@mail.gmail.com>
 <mailman.44.1478369372.7975.python-list@python.org>
 <581e29fc$0$840$e4fe514c@news.xs4all.nl>
Message-ID: <CACL+1atCX-GC+6WZnSuR13=cfkp11+ku3Z8TUaY4aUwULMikZQ@mail.gmail.com>

On Sat, Nov 5, 2016 at 6:50 PM, Irmen de Jong <irmen.NOSPAM at xs4all.nl> wrote:
> Perhaps. But in those cases you could just leave things on the default.
> If you choose to run the interpreter with eval (and exec) disabled, you should be aware
> that you'll break tools like that. But for other situations (web server etc) it could
> still be useful? I do agree that not being able to use namedtuple (and perhaps other
> things from the stdlib) is a problem then.

Breaking importlib at startup is not an option. An application would
need to import everything before disabling exec.


From jon+usenet at unequivocal.eu  Sat Nov  5 16:55:11 2016
From: jon+usenet at unequivocal.eu (Jon Ribbens)
Date: Sat, 5 Nov 2016 20:55:11 -0000 (UTC)
Subject: pip3 : command not found
References: <CAFpV4RpSbFLcBG4=Ax9jXzKEzD1Dg9N+bkST3VG-3NUtHS9Y4g@mail.gmail.com>
 <85zill9wv5.fsf@benfinney.id.au>
 <mailman.13.1477874313.17234.python-list@python.org>
 <slrno1dvok.2d7.jon+usenet@sable.unequivocal.eu>
 <5817231d$0$1593$c3e8da3$5496439d@news.astraweb.com>
 <slrno1rai3.sd6.jon+usenet@sable.unequivocal.eu>
 <581dc12e$0$1617$c3e8da3$5496439d@news.astraweb.com>
 <slrno1s095.sd6.jon+usenet@sable.unequivocal.eu>
 <581e0e81$0$1598$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <slrno1shqa.sd6.jon+usenet@sable.unequivocal.eu>

On 2016-11-05, Steve D'Aprano <steve+python at pearwood.info> wrote:
> On Sun, 6 Nov 2016 02:55 am, Jon Ribbens wrote:
>> I'm afraid I can only suggest that you try re-reading the subthread
>> again until you manage to understand it. It wasn't really that
>> complicated but you seem to have confused yourself greatly.
>
> Are you serious?

Yes. You haven't understood the thread, and I don't know how to
explain it any simpler, so I'm afraid all I can suggest is that
you re-read what was already written.

> I can only repeat yet again: you have not understood Ben.

You're still wrong no matter how many times you repeat yourself.
I've understood him, you have understood neither him nor me.
You've got hung up on the pip3 issue which is blinding you to
the other implications of Ben's answer.

Threads can and usually do diverge to a greater or lesser degree. Just
because a post is a follow-up in a thread doesn't mean that post is
entirely and solely about the original topic of that thread. Try
dropping your preconceptions and reading the sub-thread again from the
post before my first post without making false assumptions and while
bearing in mind that I am asking a question that is tangential to the
original question rather than directly descended from it.

> The implication is that the answer to your question is Yes, you can run
> Python in the context of a virtualenv by just invoking that virtualenv's
> local Python without running 'activate' first.

So you were wrong earlier when you said you couldn't do that?

You're spending an awful lot of effort being pointlessly argumentative
about who has or hasn't understood what while avoiding addressing the
actual perfectly simple question - if running Python directly from a
venv path is equivalent to running 'activate' then Python, how does it
know that it's in a venv? (Yes, I know it can sort-of work out where
it's run from using argv[0] but how does it know it's *in a venv*?)


From ben.usenet at bsb.me.uk  Sat Nov  5 17:17:35 2016
From: ben.usenet at bsb.me.uk (Ben Bacarisse)
Date: Sat, 05 Nov 2016 21:17:35 +0000
Subject: [Theory] How to speed up python code execution / pypy vs GPU
References: <nvl3q7$vut$1@news.icm.edu.pl>
 <581e25b4$0$1607$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <87pom9eids.fsf@bsb.me.uk>

Steve D'Aprano <steve+python at pearwood.info> writes:

> On Sun, 6 Nov 2016 04:10 am, Mr. Wrobel wrote:
>
>> Hi,
>> 
>> Some skeptics asked my why there is a reason to use Python against of
>> any other "not interpreted" languages, like objective-C.
>
> Here's the "Hello World" program in Python:
>
> --- cut ---
>
> print("Hello World")
>
> --- cut ---
>
>
>
> Here's the same program in Objective C:
>
> --- cut ---
>
> #import <Foundation/Foundation.h>
>
> int main (int argc, const char * argv[])
> {
>         NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
>         NSLog (@"Hello, World!");
>         [pool drain];
>         return 0;
> }
>
> --- cut ---
>
> Which would you rather write?

That's a rather odd comparison.  Why not

  #import <stdio.h>

  int main()
  {
    printf("Hello world\n");
    return 0;
  }

?  It's decades since I wrote any Objective-C (and then not much) but I
think this is the closest comparison.

-- 
Ben.


From mr at e-wrobel.pl  Sat Nov  5 18:17:55 2016
From: mr at e-wrobel.pl (Mr. Wrobel)
Date: Sat, 5 Nov 2016 23:17:55 +0100
Subject: [Theory] How to speed up python code execution / pypy vs GPU
In-Reply-To: <87pom9eids.fsf@bsb.me.uk>
References: <nvl3q7$vut$1@news.icm.edu.pl>
 <581e25b4$0$1607$c3e8da3$5496439d@news.astraweb.com>
 <87pom9eids.fsf@bsb.me.uk>
Message-ID: <nvllqc$1mf$1@news.icm.edu.pl>

W dniu 05.11.2016 o 22:17, Ben Bacarisse pisze:
> Steve D'Aprano <steve+python at pearwood.info> writes:
>
>> On Sun, 6 Nov 2016 04:10 am, Mr. Wrobel wrote:
>>
>>> Hi,
>>>
>>> Some skeptics asked my why there is a reason to use Python against of
>>> any other "not interpreted" languages, like objective-C.
>>
>> Here's the "Hello World" program in Python:
>>
>> --- cut ---
>>
>> print("Hello World")
>>
>> --- cut ---
>>
>>
>>
>> Here's the same program in Objective C:
>>
>> --- cut ---
>>
>> #import <Foundation/Foundation.h>
>>
>> int main (int argc, const char * argv[])
>> {
>>         NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
>>         NSLog (@"Hello, World!");
>>         [pool drain];
>>         return 0;
>> }
>>
>> --- cut ---
>>
>> Which would you rather write?
>
> That's a rather odd comparison.  Why not
>
>   #import <stdio.h>
>
>   int main()
>   {
>     printf("Hello world\n");
>     return 0;
>   }
>
> ?  It's decades since I wrote any Objective-C (and then not much) but I
> think this is the closest comparison.
>
Wel, indeed. However the most important is second part of my question.

What do you think about using GPU processing or pypy?


From ben.usenet at bsb.me.uk  Sat Nov  5 19:21:48 2016
From: ben.usenet at bsb.me.uk (Ben Bacarisse)
Date: Sat, 05 Nov 2016 23:21:48 +0000
Subject: [Theory] How to speed up python code execution / pypy vs GPU
References: <nvl3q7$vut$1@news.icm.edu.pl>
 <581e25b4$0$1607$c3e8da3$5496439d@news.astraweb.com>
 <87pom9eids.fsf@bsb.me.uk> <nvllqc$1mf$1@news.icm.edu.pl>
Message-ID: <878tsxecmr.fsf@bsb.me.uk>

"Mr. Wrobel" <mr at e-wrobel.pl> writes:
<snip>
> ... However the most important is second part of my question.
>
> What do you think about using GPU processing or pypy?

Sorry, I don't have enough experience of them to offer any useful advice.

-- 
Ben.


From ben+python at benfinney.id.au  Sat Nov  5 19:36:55 2016
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sun, 06 Nov 2016 10:36:55 +1100
Subject: pip3 : command not found
References: <CAFpV4RpSbFLcBG4=Ax9jXzKEzD1Dg9N+bkST3VG-3NUtHS9Y4g@mail.gmail.com>
 <85zill9wv5.fsf@benfinney.id.au>
 <mailman.13.1477874313.17234.python-list@python.org>
 <slrno1dvok.2d7.jon+usenet@sable.unequivocal.eu>
 <5817231d$0$1593$c3e8da3$5496439d@news.astraweb.com>
 <slrno1rai3.sd6.jon+usenet@sable.unequivocal.eu>
 <581dc12e$0$1617$c3e8da3$5496439d@news.astraweb.com>
 <slrno1s095.sd6.jon+usenet@sable.unequivocal.eu>
 <581e0e81$0$1598$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <8560o18pns.fsf@benfinney.id.au>

Steve D'Aprano <steve+python at pearwood.info> writes:

> Oh, and ironically, you'll see that Ben anticipated and answered your
> question about activate. Here's the link again, to save you scrolling up:
>
> https://mail.python.org/pipermail/python-list/2016-October/715994.html
>
> where he says:
>
>     If you already have a specific environment active and know that
>     ?python3? is the correct Python interpreter from that environment,
>     you can omit the explicit path.

That's what I said, and Steven is reading its meaning correctly.

-- 
 \       ?You can be a theist, and you can be a skeptic. But if you're |
  `\      both, you're not very good at one of them.? ?Dave Silverman, |
_o__)                                                       2011-11-19 |
Ben Finney



From steve+python at pearwood.info  Sat Nov  5 20:52:51 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sun, 06 Nov 2016 11:52:51 +1100
Subject: [Theory] How to speed up python code execution / pypy vs GPU
References: <nvl3q7$vut$1@news.icm.edu.pl>
 <581e25b4$0$1607$c3e8da3$5496439d@news.astraweb.com>
 <87pom9eids.fsf@bsb.me.uk>
Message-ID: <581e7ee6$0$1608$c3e8da3$5496439d@news.astraweb.com>

On Sun, 6 Nov 2016 08:17 am, Ben Bacarisse wrote:

> Steve D'Aprano <steve+python at pearwood.info> writes:

>> Here's the same program in Objective C:
>>
>> --- cut ---
>>
>> #import <Foundation/Foundation.h>
>>
>> int main (int argc, const char * argv[])
>> {
>>         NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
>>         NSLog (@"Hello, World!");
>>         [pool drain];
>>         return 0;
>> }
>>
>> --- cut ---
>>
>> Which would you rather write?
> 
> That's a rather odd comparison.  Why not
> 
>   #import <stdio.h>
> 
>   int main()
>   {
>     printf("Hello world\n");
>     return 0;
>   }


Because that's not Objective-C? (This is not a rhetorical question.)

I'm not an Objective-C expert, but to my eye, that doesn't look like
Objective-C. It looks like plain old regular C.

Here's where I stole the code from:

https://www.binpress.com/tutorial/objectivec-lesson-1-hello-world/41

and its not too dissimilar from the versions here:

http://rosettacode.org/wiki/Hello_world/Text#Objective-C


> ?  It's decades since I wrote any Objective-C (and then not much) but I
> think this is the closest comparison.



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Sat Nov  5 21:32:06 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sun, 06 Nov 2016 12:32:06 +1100
Subject: pip3 : command not found
References: <CAFpV4RpSbFLcBG4=Ax9jXzKEzD1Dg9N+bkST3VG-3NUtHS9Y4g@mail.gmail.com>
 <85zill9wv5.fsf@benfinney.id.au>
 <mailman.13.1477874313.17234.python-list@python.org>
 <slrno1dvok.2d7.jon+usenet@sable.unequivocal.eu>
 <5817231d$0$1593$c3e8da3$5496439d@news.astraweb.com>
 <slrno1rai3.sd6.jon+usenet@sable.unequivocal.eu>
 <581dc12e$0$1617$c3e8da3$5496439d@news.astraweb.com>
 <slrno1s095.sd6.jon+usenet@sable.unequivocal.eu>
 <581e0e81$0$1598$c3e8da3$5496439d@news.astraweb.com>
 <slrno1shqa.sd6.jon+usenet@sable.unequivocal.eu>
Message-ID: <581e8817$0$1616$c3e8da3$5496439d@news.astraweb.com>

On Sun, 6 Nov 2016 07:55 am, Jon Ribbens wrote:

>> The implication is that the answer to your question is Yes, you can run
>> Python in the context of a virtualenv by just invoking that virtualenv's
>> local Python without running 'activate' first.
> 
> So you were wrong earlier when you said you couldn't do that?

I'm happy to admit it when I have made a mistake, but this isn't one of
those times. I didn't say it couldn't be done, I initially said:

    "Possibly you do have to run `activate` first, I don't know"

but you already know that, since you can read just as well as I can.



> You're still wrong no matter how many times you repeat yourself.
> I've understood him, you have understood neither him nor me.
[...]
> You're spending an awful lot of effort being pointlessly argumentative
> about who has or hasn't understood what while avoiding addressing the
> actual perfectly simple question - if running Python directly from a
> venv path is equivalent to running 'activate' then Python, how does it
> know that it's in a venv?

And now you change your story for the second time.

As I said, I'm happy to admit when I am wrong. I was wrong to think you were
discussing this in good faith. You're clearly not: you keep changing your
story, keep insisting that the question you have asked is different from
what you previously wrong.

First you asked about symlinks, misunderstanding what Ben wrote; then you
denied you asked that and insisted you really asked about activate (despite
not having even mentioned activate); and now you're inventing a new
question, "how does it know it is in a venv?".

Life is too short to waste time in a pointless discussion with trolls who
argue in bad faith, as you are doing. Welcome to my kill-file.

*plonk*





-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Sat Nov  5 21:39:40 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sun, 06 Nov 2016 12:39:40 +1100
Subject: [Theory] How to speed up python code execution / pypy vs GPU
References: <nvl3q7$vut$1@news.icm.edu.pl>
 <581e25b4$0$1607$c3e8da3$5496439d@news.astraweb.com>
 <87pom9eids.fsf@bsb.me.uk> <nvllqc$1mf$1@news.icm.edu.pl>
Message-ID: <581e89dd$0$1620$c3e8da3$5496439d@news.astraweb.com>

On Sun, 6 Nov 2016 09:17 am, Mr. Wrobel wrote:

> However the most important is second part of my question.
> 
> What do you think about using GPU processing or pypy?

I don't have any experience with GPU processing. I expect that it will be
useful for somethings, but for number-crushing and numeric work, I am
concerned that GPUs rarely provide correctly rounded IEEE-754 maths. That
means that they are accurate enough for games where a few visual glitches
don't matter, but they risk being inaccurate for serious work.

I fear that doing numeric work in GPUs will be returning to the 1970s, when
every computer was incompatible with every other computer, and it was
almost impossible to write cross-platform, correct, accurate numeric code.

As far as PyPy goes, it is a proven optimizing Python JIT compiler that can
speed-up long-running code very well. It is not suitable for short scripts
or programmers that run quickly. It takes time for the JIT to warm up and
start showing optimizations.




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From ben.usenet at bsb.me.uk  Sat Nov  5 21:42:32 2016
From: ben.usenet at bsb.me.uk (Ben Bacarisse)
Date: Sun, 06 Nov 2016 01:42:32 +0000
Subject: [Theory] How to speed up python code execution / pypy vs GPU
References: <nvl3q7$vut$1@news.icm.edu.pl>
 <581e25b4$0$1607$c3e8da3$5496439d@news.astraweb.com>
 <87pom9eids.fsf@bsb.me.uk>
 <581e7ee6$0$1608$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <87r36pcrjr.fsf@bsb.me.uk>

Steve D'Aprano <steve+python at pearwood.info> writes:

> On Sun, 6 Nov 2016 08:17 am, Ben Bacarisse wrote:
>
>> Steve D'Aprano <steve+python at pearwood.info> writes:
>
>>> Here's the same program in Objective C:
>>>
>>> --- cut ---
>>>
>>> #import <Foundation/Foundation.h>
>>>
>>> int main (int argc, const char * argv[])
>>> {
>>>         NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
>>>         NSLog (@"Hello, World!");
>>>         [pool drain];
>>>         return 0;
>>> }
>>>
>>> --- cut ---
>>>
>>> Which would you rather write?
>> 
>> That's a rather odd comparison.  Why not
>> 
>>   #import <stdio.h>
>> 
>>   int main()
>>   {
>>     printf("Hello world\n");
>>     return 0;
>>   }
>
> Because that's not Objective-C? (This is not a rhetorical question.)

Yes, I think so.

> I'm not an Objective-C expert, but to my eye, that doesn't look like
> Objective-C. It looks like plain old regular C.

C has no #import directive.

<snip>
-- 
Ben.


From rustompmody at gmail.com  Sat Nov  5 23:22:46 2016
From: rustompmody at gmail.com (Rustom Mody)
Date: Sat, 5 Nov 2016 20:22:46 -0700 (PDT)
Subject: distributed development methodology
In-Reply-To: <878tt7mw5m.fsf@nightsong.com>
References: <nv13et$dp5$1@dont-email.me> <878tt7mw5m.fsf@nightsong.com>
Message-ID: <f15a7380-ff63-4add-a5d4-7d5ff5195f97@googlegroups.com>

On Saturday, October 29, 2016 at 1:20:25 PM UTC+5:30, Paul Rubin wrote:
> Adam Jensen  writes:
> > So what are some of the more successful distributed. multi-platform,
> > development models?
> 
> Use an orchestration program to keep the systems in sync: I use ansible
> (ansible.com) which is written in Python and fairly simple once you get
> used to it, but there are lots of other programs in that space and
> everyone has their own preferences.
> 
> If stuff doesn't change too often, you could also use Docker or whatever
> the current hotness is to make a container image and deploy it to your
> fleet.

Been seeing things like this of late:
https://thehftguy.wordpress.com/2016/11/01/docker-in-production-an-history-of-failure/
[Disclaimer: No direct experience myself one way or other]

LXC and others compared: https://www.flockport.com/alternatives-to-docker-and-lxc/


From jon+usenet at unequivocal.eu  Sun Nov  6 00:03:40 2016
From: jon+usenet at unequivocal.eu (Jon Ribbens)
Date: Sun, 6 Nov 2016 04:03:40 -0000 (UTC)
Subject: pip3 : command not found
References: <CAFpV4RpSbFLcBG4=Ax9jXzKEzD1Dg9N+bkST3VG-3NUtHS9Y4g@mail.gmail.com>
 <85zill9wv5.fsf@benfinney.id.au>
 <mailman.13.1477874313.17234.python-list@python.org>
 <slrno1dvok.2d7.jon+usenet@sable.unequivocal.eu>
 <5817231d$0$1593$c3e8da3$5496439d@news.astraweb.com>
 <slrno1rai3.sd6.jon+usenet@sable.unequivocal.eu>
 <581dc12e$0$1617$c3e8da3$5496439d@news.astraweb.com>
 <slrno1s095.sd6.jon+usenet@sable.unequivocal.eu>
 <581e0e81$0$1598$c3e8da3$5496439d@news.astraweb.com>
 <slrno1shqa.sd6.jon+usenet@sable.unequivocal.eu>
 <581e8817$0$1616$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <slrno1tato.sd6.jon+usenet@sable.unequivocal.eu>

On 2016-11-06, Steve D'Aprano <steve+python at pearwood.info> wrote:
> *plonk*

Thank feck for that, I was beginning to think he'd never shut up.

I don't suppose anyone else more constructive and informed actually
knows the answer to my rather simple question of how Python knows
it's in a venv? ;-)


From rosuav at gmail.com  Sun Nov  6 00:23:22 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 6 Nov 2016 15:23:22 +1100
Subject: pip3 : command not found
In-Reply-To: <slrno1tato.sd6.jon+usenet@sable.unequivocal.eu>
References: <CAFpV4RpSbFLcBG4=Ax9jXzKEzD1Dg9N+bkST3VG-3NUtHS9Y4g@mail.gmail.com>
 <85zill9wv5.fsf@benfinney.id.au>
 <mailman.13.1477874313.17234.python-list@python.org>
 <slrno1dvok.2d7.jon+usenet@sable.unequivocal.eu>
 <5817231d$0$1593$c3e8da3$5496439d@news.astraweb.com>
 <slrno1rai3.sd6.jon+usenet@sable.unequivocal.eu>
 <581dc12e$0$1617$c3e8da3$5496439d@news.astraweb.com>
 <slrno1s095.sd6.jon+usenet@sable.unequivocal.eu>
 <581e0e81$0$1598$c3e8da3$5496439d@news.astraweb.com>
 <slrno1shqa.sd6.jon+usenet@sable.unequivocal.eu>
 <581e8817$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <slrno1tato.sd6.jon+usenet@sable.unequivocal.eu>
Message-ID: <CAPTjJmqHuYGhC77_mPzztqv8b1_gm_1LZDfPn3StSvJfo0CDag@mail.gmail.com>

On Sun, Nov 6, 2016 at 3:03 PM, Jon Ribbens <jon+usenet at unequivocal.eu> wrote:
> I don't suppose anyone else more constructive and informed actually
> knows the answer to my rather simple question of how Python knows
> it's in a venv? ;-)

Two ways.

1) Normally, you 'activate' the venv by sourcing a script into your
shell. This modifies $PATH, $PYTHONHOME, and I think a couple of other
environment variables.

2) If Python notices that its executable comes from a venv, it uses it.

(I wasn't aware of #2 until this thread.)

Generally, you type "python" or "python3" and the shell searches
$PATH. If you have a venv active, it'll find the one there before it
finds the system one.

ChrisA


From jon+usenet at unequivocal.eu  Sun Nov  6 01:27:10 2016
From: jon+usenet at unequivocal.eu (Jon Ribbens)
Date: Sun, 6 Nov 2016 05:27:10 -0000 (UTC)
Subject: pip3 : command not found
References: <CAFpV4RpSbFLcBG4=Ax9jXzKEzD1Dg9N+bkST3VG-3NUtHS9Y4g@mail.gmail.com>
 <85zill9wv5.fsf@benfinney.id.au>
 <mailman.13.1477874313.17234.python-list@python.org>
 <slrno1dvok.2d7.jon+usenet@sable.unequivocal.eu>
 <5817231d$0$1593$c3e8da3$5496439d@news.astraweb.com>
 <slrno1rai3.sd6.jon+usenet@sable.unequivocal.eu>
 <581dc12e$0$1617$c3e8da3$5496439d@news.astraweb.com>
 <slrno1s095.sd6.jon+usenet@sable.unequivocal.eu>
 <581e0e81$0$1598$c3e8da3$5496439d@news.astraweb.com>
 <slrno1shqa.sd6.jon+usenet@sable.unequivocal.eu>
 <581e8817$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <slrno1tato.sd6.jon+usenet@sable.unequivocal.eu>
 <CAPTjJmqHuYGhC77_mPzztqv8b1_gm_1LZDfPn3StSvJfo0CDag@mail.gmail.com>
 <mailman.0.1478406205.30068.python-list@python.org>
Message-ID: <slrno1tfqa.sd6.jon+usenet@sable.unequivocal.eu>

On 2016-11-06, Chris Angelico <rosuav at gmail.com> wrote:
> On Sun, Nov 6, 2016 at 3:03 PM, Jon Ribbens <jon+usenet at unequivocal.eu> wrote:
>> I don't suppose anyone else more constructive and informed actually
>> knows the answer to my rather simple question of how Python knows
>> it's in a venv? ;-)
>
> Two ways.
>
> 1) Normally, you 'activate' the venv by sourcing a script into your
> shell. This modifies $PATH, $PYTHONHOME, and I think a couple of other
> environment variables.

It sets VIRTUAL_ENV, adds the virtualenv's bin directory to PATH,
and *unsets* PYTHONHOME if set. That's it (modulo storing old values
of variables and updating PS1 which is presumably just cosmetic).

> 2) If Python notices that its executable comes from a venv, it uses it.

Yes. My question is *how does it notice*?


From rosefox911 at gmail.com  Sun Nov  6 01:27:24 2016
From: rosefox911 at gmail.com (rosefox911 at gmail.com)
Date: Sat, 5 Nov 2016 22:27:24 -0700 (PDT)
Subject: Delete h2 until you reach the next h2 in beautifulsoup
Message-ID: <c9c99de5-3b80-45b4-8e14-509fef08f3d5@googlegroups.com>

Considering the following html:

    <h2 id="example">cool stuff</h2> <ul> <li>hi</li> </ul> <div> <h2 id="cool"><h2> <ul><li>zz</li> </ul> </div>

and the following list:

    ignore_list = ['example','lalala']

My goal is, while going through the HTML using Beautifulsoup, I find a h2 that has an ID that is in my list (ignore_list) I should delete all the ul and lis under it until I find another h2. I would then check if the next h2 was in my ignore list, if it is, delete all the ul and lis until I reach the next h2 (or if there are no h2s left, delete the ul and lis under the current one and stop). 

How I see the process going: you read all the h2s from up to down in the DOM. If the id for any of those is in the ignore_list, then delete all the ul and li under the h2 until you reach the NEXT h2. If there is no h2, then delete the ul and LI then stop.

Here is the full HMTL I am trying to work with: http://pastebin.com/Z3ev9c8N

I am trying to delete all the UL and lis after "See_also"How would I accomplish this in Python?


From ben+python at benfinney.id.au  Sun Nov  6 01:27:59 2016
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sun, 06 Nov 2016 16:27:59 +1100
Subject: pip3 : command not found
References: <CAFpV4RpSbFLcBG4=Ax9jXzKEzD1Dg9N+bkST3VG-3NUtHS9Y4g@mail.gmail.com>
 <85zill9wv5.fsf@benfinney.id.au>
 <mailman.13.1477874313.17234.python-list@python.org>
 <slrno1dvok.2d7.jon+usenet@sable.unequivocal.eu>
 <5817231d$0$1593$c3e8da3$5496439d@news.astraweb.com>
 <slrno1rai3.sd6.jon+usenet@sable.unequivocal.eu>
 <581dc12e$0$1617$c3e8da3$5496439d@news.astraweb.com>
 <slrno1s095.sd6.jon+usenet@sable.unequivocal.eu>
 <581e0e81$0$1598$c3e8da3$5496439d@news.astraweb.com>
 <slrno1shqa.sd6.jon+usenet@sable.unequivocal.eu>
 <581e8817$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <slrno1tato.sd6.jon+usenet@sable.unequivocal.eu>
Message-ID: <85wpgh6uu8.fsf@benfinney.id.au>

Jon Ribbens <jon+usenet at unequivocal.eu> writes:

> On 2016-11-06, Steve D'Aprano <steve+python at pearwood.info> wrote:
> > *plonk*
>
> Thank feck for that, I was beginning to think he'd never shut up.

Really? I didn't see a single message from Steven that wasn't solicited
by an assertion from you that needed response.

If your concept of ?won't shut up? includes ?responds to me when I
demand he acknowledge my position is correct?, then it seems clear the
problem has a different solution.

-- 
 \           ?Prayer must never be answered: if it is, it ceases to be |
  `\   prayer and becomes correspondence.? ?Oscar Wilde, _The Epigrams |
_o__)                                            of Oscar Wilde_, 1952 |
Ben Finney



From jon+usenet at unequivocal.eu  Sun Nov  6 01:41:53 2016
From: jon+usenet at unequivocal.eu (Jon Ribbens)
Date: Sun, 6 Nov 2016 05:41:53 -0000 (UTC)
Subject: pip3 : command not found
References: <CAFpV4RpSbFLcBG4=Ax9jXzKEzD1Dg9N+bkST3VG-3NUtHS9Y4g@mail.gmail.com>
 <85zill9wv5.fsf@benfinney.id.au>
 <mailman.13.1477874313.17234.python-list@python.org>
 <slrno1dvok.2d7.jon+usenet@sable.unequivocal.eu>
 <5817231d$0$1593$c3e8da3$5496439d@news.astraweb.com>
 <slrno1rai3.sd6.jon+usenet@sable.unequivocal.eu>
 <581dc12e$0$1617$c3e8da3$5496439d@news.astraweb.com>
 <slrno1s095.sd6.jon+usenet@sable.unequivocal.eu>
 <581e0e81$0$1598$c3e8da3$5496439d@news.astraweb.com>
 <slrno1shqa.sd6.jon+usenet@sable.unequivocal.eu>
 <581e8817$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <slrno1tato.sd6.jon+usenet@sable.unequivocal.eu>
 <85wpgh6uu8.fsf@benfinney.id.au>
 <mailman.1.1478410113.30068.python-list@python.org>
Message-ID: <slrno1tglt.sd6.jon+usenet@sable.unequivocal.eu>

On 2016-11-06, Ben Finney <ben+python at benfinney.id.au> wrote:
> Jon Ribbens <jon+usenet at unequivocal.eu> writes:
>> On 2016-11-06, Steve D'Aprano <steve+python at pearwood.info> wrote:
>> > *plonk*
>>
>> Thank feck for that, I was beginning to think he'd never shut up.
>
> Really? I didn't see a single message from Steven that wasn't solicited
> by an assertion from you that needed response.
>
> If your concept of ?won't shut up? includes ?responds to me when I
> demand he acknowledge my position is correct?, then it seems clear the
> problem has a different solution.

He consistently misunderstood almost everything I said (which is
a pattern of behaviour he seems to have, i.e. heroically
misunderstanding people so he can argue with them), lied about
what I had said, lied about what he had said, lied about me "not
arguing in good faith", was rude and insulting, and studiously avoided
the actual interesting and useful topic in favour of a stupid and
pointless flamewar about his misunderstanding of my original question.

He clearly didn't know the answer to the question nor did he have any
interest in dicussing it in a civilised manner with a view to
discovering the answer, he just wants an argument. So yes, I view him
bowing out of the conversation as an unmitigated blessing.


From ben+python at benfinney.id.au  Sun Nov  6 01:50:53 2016
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sun, 06 Nov 2016 16:50:53 +1100
Subject: pip3 : command not found
References: <CAFpV4RpSbFLcBG4=Ax9jXzKEzD1Dg9N+bkST3VG-3NUtHS9Y4g@mail.gmail.com>
 <85zill9wv5.fsf@benfinney.id.au>
 <mailman.13.1477874313.17234.python-list@python.org>
 <slrno1dvok.2d7.jon+usenet@sable.unequivocal.eu>
 <5817231d$0$1593$c3e8da3$5496439d@news.astraweb.com>
 <slrno1rai3.sd6.jon+usenet@sable.unequivocal.eu>
 <581dc12e$0$1617$c3e8da3$5496439d@news.astraweb.com>
 <slrno1s095.sd6.jon+usenet@sable.unequivocal.eu>
 <581e0e81$0$1598$c3e8da3$5496439d@news.astraweb.com>
 <slrno1shqa.sd6.jon+usenet@sable.unequivocal.eu>
 <581e8817$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <slrno1tato.sd6.jon+usenet@sable.unequivocal.eu>
 <85wpgh6uu8.fsf@benfinney.id.au>
 <mailman.1.1478410113.30068.python-list@python.org>
 <slrno1tglt.sd6.jon+usenet@sable.unequivocal.eu>
Message-ID: <85shr56ts2.fsf@benfinney.id.au>

Jon Ribbens <jon+usenet at unequivocal.eu> writes:

> He [?] lied about me "not arguing in good faith"

I find you to be not arguing in good faith; if you consistently engage
someone in a manner that requires response, that's not consistent with
also ?wondering when they'd shut up?.

> So yes, I view him bowing out of the conversation as an unmitigated
> blessing.

You could have disengaged at any time, so I don't find your statement
believable.

-- 
 \       ?Firmness in decision is often merely a form of stupidity. It |
  `\        indicates an inability to think the same thing out twice.? |
_o__)                                                ?Henry L. Mencken |
Ben Finney



From jon+usenet at unequivocal.eu  Sun Nov  6 01:03:11 2016
From: jon+usenet at unequivocal.eu (Jon Ribbens)
Date: Sun, 6 Nov 2016 06:03:11 -0000 (UTC)
Subject: pip3 : command not found
References: <CAFpV4RpSbFLcBG4=Ax9jXzKEzD1Dg9N+bkST3VG-3NUtHS9Y4g@mail.gmail.com>
 <85zill9wv5.fsf@benfinney.id.au>
 <mailman.13.1477874313.17234.python-list@python.org>
 <slrno1dvok.2d7.jon+usenet@sable.unequivocal.eu>
 <5817231d$0$1593$c3e8da3$5496439d@news.astraweb.com>
 <slrno1rai3.sd6.jon+usenet@sable.unequivocal.eu>
 <581dc12e$0$1617$c3e8da3$5496439d@news.astraweb.com>
 <slrno1s095.sd6.jon+usenet@sable.unequivocal.eu>
 <581e0e81$0$1598$c3e8da3$5496439d@news.astraweb.com>
 <slrno1shqa.sd6.jon+usenet@sable.unequivocal.eu>
 <581e8817$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <slrno1tato.sd6.jon+usenet@sable.unequivocal.eu>
 <85wpgh6uu8.fsf@benfinney.id.au>
 <mailman.1.1478410113.30068.python-list@python.org>
 <slrno1tglt.sd6.jon+usenet@sable.unequivocal.eu>
 <85shr56ts2.fsf@benfinney.id.au>
 <mailman.2.1478411516.30068.python-list@python.org>
Message-ID: <slrno1thtr.sd6.jon+usenet@sable.unequivocal.eu>

On 2016-11-06, Ben Finney <ben+python at benfinney.id.au> wrote:
> Jon Ribbens <jon+usenet at unequivocal.eu> writes:
>
>> He [?] lied about me "not arguing in good faith"
>
> I find you to be not arguing in good faith;

I find that to be particularly pompous of you. I am arguing in good
faith regardless of your misguided opinion.

I wasn't actually even trying to argue at all, I simply had
a question, which Steven successfully derailed into a stupid argument
as to whether he knew better than me what question I was asking.

>> So yes, I view him bowing out of the conversation as an unmitigated
>> blessing.
>
> You could have disengaged at any time, so I don't find your statement
> believable.

I plead guilty to an excess of optimism that I could persuade him to
start discussing the actual topic. I had actually reached the end of
my optimism in that regard and the last post I had made to him was
the last one in which I had intended to bother trying to get through
to him.


From alec.taylor6 at gmail.com  Sun Nov  6 05:17:16 2016
From: alec.taylor6 at gmail.com (Alec Taylor)
Date: Sun, 6 Nov 2016 21:17:16 +1100
Subject: Force virtualenv pip to be used
Message-ID: <CAO+9iGcW_u9f0i00ZrVHOnOQf5bFMGzkk_dfhEesr6f8gsh=fQ@mail.gmail.com>

Running Ubuntu 16.10 with Python 2.7.12+ (default one) and virtualenv
15.0.3 (`sudo -H pip install virtualenv`). What am I doing wrong?

    $ virtualenv a && . "$_"/bin/activate && pip --version
    New python executable in /tmp/a/bin/python
    Installing setuptools, pip, wheel...done.
    pip 9.0.0 from /usr/local/lib/python2.7/dist-packages (python 2.7)
    $ /tmp/a/bin/pip --version
    pip 9.0.0 from /usr/local/lib/python2.7/dist-packages (python 2.7)
    $ /tmp/a/bin/python -c 'from pip import __file__; print __file__'
    /usr/local/lib/python2.7/dist-packages/pip/__init__.pyc

PS: Running with regular Bash 4.3.46(1)-release in GNOME Terminal. When I
activate the virtualenv this appears in my env output: VIRTUAL_ENV=/tmp/a

PPS: Cross-posted [yesterday] http://stackoverflow.com/q/40438089


From __peter__ at web.de  Sun Nov  6 05:45:14 2016
From: __peter__ at web.de (Peter Otten)
Date: Sun, 06 Nov 2016 11:45:14 +0100
Subject: Force virtualenv pip to be used
References: <CAO+9iGcW_u9f0i00ZrVHOnOQf5bFMGzkk_dfhEesr6f8gsh=fQ@mail.gmail.com>
Message-ID: <nvn1jr$ajj$1@blaine.gmane.org>

Alec Taylor wrote:

> Running Ubuntu 16.10 with Python 2.7.12+ (default one) and virtualenv
> 15.0.3 (`sudo -H pip install virtualenv`). What am I doing wrong?
> 
>     $ virtualenv a && . "$_"/bin/activate && pip --version
>     New python executable in /tmp/a/bin/python
>     Installing setuptools, pip, wheel...done.
>     pip 9.0.0 from /usr/local/lib/python2.7/dist-packages (python 2.7)
>     $ /tmp/a/bin/pip --version
>     pip 9.0.0 from /usr/local/lib/python2.7/dist-packages (python 2.7)
>     $ /tmp/a/bin/python -c 'from pip import __file__; print __file__'
>     /usr/local/lib/python2.7/dist-packages/pip/__init__.pyc
> 
> PS: Running with regular Bash 4.3.46(1)-release in GNOME Terminal. When I
> activate the virtualenv this appears in my env output: VIRTUAL_ENV=/tmp/a
> 
> PPS: Cross-posted [yesterday] http://stackoverflow.com/q/40438089

What's in your PYTHONPATH?

$ export PYTHONPATH=/usr/lib/python2.7/dist-packages/
$ virtualenv a && . "$_"/bin/activate && pip --version
New python executable in a/bin/python
Installing setuptools, pip...done.
pip 1.5.4 from /usr/lib/python2.7/dist-packages (python 2.7)
(a)$ unset PYTHONPATH
(a)$ pip --version
pip 1.5.4 from /home/peter/a/local/lib/python2.7/site-packages (python 2.7)




From rosuav at gmail.com  Sun Nov  6 06:10:11 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 6 Nov 2016 22:10:11 +1100
Subject: Force virtualenv pip to be used
In-Reply-To: <CAO+9iGcW_u9f0i00ZrVHOnOQf5bFMGzkk_dfhEesr6f8gsh=fQ@mail.gmail.com>
References: <CAO+9iGcW_u9f0i00ZrVHOnOQf5bFMGzkk_dfhEesr6f8gsh=fQ@mail.gmail.com>
Message-ID: <CAPTjJmp94ikiureFbkQaz81+LbDiBLpnxWGeerC+DAzGZgmkMA@mail.gmail.com>

On Sun, Nov 6, 2016 at 9:17 PM, Alec Taylor <alec.taylor6 at gmail.com> wrote:
> Running Ubuntu 16.10 with Python 2.7.12+ (default one) and virtualenv
> 15.0.3 (`sudo -H pip install virtualenv`). What am I doing wrong?
>
>     $ virtualenv a && . "$_"/bin/activate && pip --version

I'm pretty sure virtualenv (like venv, about which I'm certain)
creates something that you have to 'source' into your shell, rather
than running in the classic way:

source env/bin/activate

It needs to alter environment variables in your shell, which can't be
done from a separate program.

ChrisA


From rosuav at gmail.com  Sun Nov  6 06:13:12 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 6 Nov 2016 22:13:12 +1100
Subject: pip3 : command not found
In-Reply-To: <slrno1tfqa.sd6.jon+usenet@sable.unequivocal.eu>
References: <CAFpV4RpSbFLcBG4=Ax9jXzKEzD1Dg9N+bkST3VG-3NUtHS9Y4g@mail.gmail.com>
 <85zill9wv5.fsf@benfinney.id.au>
 <mailman.13.1477874313.17234.python-list@python.org>
 <slrno1dvok.2d7.jon+usenet@sable.unequivocal.eu>
 <5817231d$0$1593$c3e8da3$5496439d@news.astraweb.com>
 <slrno1rai3.sd6.jon+usenet@sable.unequivocal.eu>
 <581dc12e$0$1617$c3e8da3$5496439d@news.astraweb.com>
 <slrno1s095.sd6.jon+usenet@sable.unequivocal.eu>
 <581e0e81$0$1598$c3e8da3$5496439d@news.astraweb.com>
 <slrno1shqa.sd6.jon+usenet@sable.unequivocal.eu>
 <581e8817$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <slrno1tato.sd6.jon+usenet@sable.unequivocal.eu>
 <CAPTjJmqHuYGhC77_mPzztqv8b1_gm_1LZDfPn3StSvJfo0CDag@mail.gmail.com>
 <mailman.0.1478406205.30068.python-list@python.org>
 <slrno1tfqa.sd6.jon+usenet@sable.unequivocal.eu>
Message-ID: <CAPTjJmrs+LtPh44o43xM8v4rjgH9MXneQ-zSuj9Y1iusvvqCRQ@mail.gmail.com>

On Sun, Nov 6, 2016 at 4:27 PM, Jon Ribbens <jon+usenet at unequivocal.eu> wrote:
>> 2) If Python notices that its executable comes from a venv, it uses it.
>
> Yes. My question is *how does it notice*?

I could answer this question, but since you don't appear to be
following the thread properly, I'll point out that it's already been
said up above. Go read it.

ChrisA


From __peter__ at web.de  Sun Nov  6 06:19:27 2016
From: __peter__ at web.de (Peter Otten)
Date: Sun, 06 Nov 2016 12:19:27 +0100
Subject: Force virtualenv pip to be used
References: <CAO+9iGcW_u9f0i00ZrVHOnOQf5bFMGzkk_dfhEesr6f8gsh=fQ@mail.gmail.com>
 <CAPTjJmp94ikiureFbkQaz81+LbDiBLpnxWGeerC+DAzGZgmkMA@mail.gmail.com>
Message-ID: <nvn3k3$t94$1@blaine.gmane.org>

Chris Angelico wrote:

> On Sun, Nov 6, 2016 at 9:17 PM, Alec Taylor <alec.taylor6 at gmail.com>
> wrote:
>> Running Ubuntu 16.10 with Python 2.7.12+ (default one) and virtualenv
>> 15.0.3 (`sudo -H pip install virtualenv`). What am I doing wrong?
>>
>>     $ virtualenv a && . "$_"/bin/activate && pip --version
> 
> I'm pretty sure virtualenv (like venv, about which I'm certain)
> creates something that you have to 'source' into your shell, rather
> than running in the classic way:
> 
> source env/bin/activate

I think this is what the

. "$_"/bin/activate

part of Alec's command is supposed to do. 

Yes, that's a dot, not grit on Tim's screen ;)

> It needs to alter environment variables in your shell, which can't be
> done from a separate program.




From alec.taylor6 at gmail.com  Sun Nov  6 06:29:25 2016
From: alec.taylor6 at gmail.com (Alec Taylor)
Date: Sun, 6 Nov 2016 03:29:25 -0800 (PST)
Subject: Force virtualenv pip to be used
In-Reply-To: <mailman.7.1478431240.30068.python-list@python.org>
References: <CAO+9iGcW_u9f0i00ZrVHOnOQf5bFMGzkk_dfhEesr6f8gsh=fQ@mail.gmail.com>
 <CAPTjJmp94ikiureFbkQaz81+LbDiBLpnxWGeerC+DAzGZgmkMA@mail.gmail.com>
 <nvn3k3$t94$1@blaine.gmane.org>
 <mailman.7.1478431240.30068.python-list@python.org>
Message-ID: <e750549b-7e9e-4664-ae20-e989b690c920@googlegroups.com>

On Sunday, November 6, 2016 at 10:20:53 PM UTC+11, Peter Otten wrote:
> Chris Angelico wrote:
> 
> > On Sun, Nov 6, 2016 at 9:17 PM, Alec Taylor <alec.taylor6 at gmail.com>
> > wrote:
> >> Running Ubuntu 16.10 with Python 2.7.12+ (default one) and virtualenv
> >> 15.0.3 (`sudo -H pip install virtualenv`). What am I doing wrong?
> >>
> >>     $ virtualenv a && . "$_"/bin/activate && pip --version
> > 
> > I'm pretty sure virtualenv (like venv, about which I'm certain)
> > creates something that you have to 'source' into your shell, rather
> > than running in the classic way:
> > 
> > source env/bin/activate
> 
> I think this is what the
> 
> . "$_"/bin/activate
> 
> part of Alec's command is supposed to do. 
> 
> Yes, that's a dot, not grit on Tim's screen ;)
> 
> > It needs to alter environment variables in your shell, which can't be
> > done from a separate program.

Hmm:
$ echo $PYTHONPATH
/usr/local/lib/python2.7/site-packages:/usr/local/lib/python2.7/dist-packages
$ pip --version
pip 9.0.0 from /tmp/a/local/lib/python2.7/site-packages (python 2.7)


So that worked. Shouldn't PYTHONPATH be set/upserted by the activation of the virtualenv?


From i at introo.me  Sun Nov  6 06:41:06 2016
From: i at introo.me (Shiyao Ma)
Date: Sun, 6 Nov 2016 03:41:06 -0800 (PST)
Subject: Regarding the parsing of await expression.
Message-ID: <b11209b9-e971-49a3-b7e1-1a5ec9abcf09@googlegroups.com>

Hi,

In the pep, https://www.python.org/dev/peps/pep-0492/#examples-of-await-expressions 

It is said,

await await coro()  is SyntaxError, instead, we should use await (await coro())
Why? because of await is not left-associative?


also, for 
await -coro() , it should be written as,  await (-coro())

I don't understand the point here.  Why can't the parser figure out it indeed is await (-coro()) ?


Is it due to the fact Python uses LL(1) or just because of current impl doesn't do that?

Regards.


From sami.strat at gmail.com  Sun Nov  6 06:48:31 2016
From: sami.strat at gmail.com (SS)
Date: Sun, 6 Nov 2016 03:48:31 -0800 (PST)
Subject: passing a variable to cmd
Message-ID: <d93c29f8-adee-4657-9f65-e1ef95208a79@googlegroups.com>

Note the following code:

import subprocess
import shlex

domname = raw_input("Enter your domain name: ");
print "Your domain name is: ", domname

print "\n"

# cmd='dig @4.2.2.2 nbc.com ns +short'
cmd="dig @4.2.2.2 %s ns +short", % (domname)
proc=subprocess.Popen(shlex.split(cmd),stdout=subprocess.PIPE)
out,err=proc.communicate()
print(out)

The line that is commented out works fine.  However, I want to substitute a variable for "nbc.com".  The command:

cmd="dig @4.2.2.2 %s ns +short", % (domname)

does not work.  I've tried different variations to no avail.  Any advice on how to get that variable working?

TIA.


From jsf80238 at gmail.com  Sun Nov  6 07:47:48 2016
From: jsf80238 at gmail.com (Jason Friedman)
Date: Sun, 6 Nov 2016 05:47:48 -0700
Subject: passing a variable to cmd
In-Reply-To: <d93c29f8-adee-4657-9f65-e1ef95208a79@googlegroups.com>
References: <d93c29f8-adee-4657-9f65-e1ef95208a79@googlegroups.com>
Message-ID: <CANy1k1jM2HVoJAnVj=VpC6em2fgPrtBnHQ7c-9h-X62zVkuJ2w@mail.gmail.com>

>
> import subprocess
> import shlex
>
> domname = raw_input("Enter your domain name: ");
> print "Your domain name is: ", domname
>
> print "\n"
>
> # cmd='dig @4.2.2.2 nbc.com ns +short'
> cmd="dig @4.2.2.2 %s ns +short", % (domname)
> proc=subprocess.Popen(shlex.split(cmd),stdout=subprocess.PIPE)
> out,err=proc.communicate()
> print(out)
>
> The line that is commented out works fine.  However, I want to substitute
> a variable for "nbc.com".  The command:
>
> cmd="dig @4.2.2.2 %s ns +short", % (domname)
>
> does not work.  I've tried different variations to no avail.  Any advice
> on how to get that variable working?
>

Two things:
1) Please explain what "does not work" mean.  Are you getting an error
message?  If yes include the entire traceback.
2) The line /* cmd="dig @4.2.2.2 %s ns +short", % (domname) */ gives me a
syntax error.  Please copy-and-paste the exact commands you are entering.


From __peter__ at web.de  Sun Nov  6 08:36:33 2016
From: __peter__ at web.de (Peter Otten)
Date: Sun, 06 Nov 2016 14:36:33 +0100
Subject: Force virtualenv pip to be used
References: <CAO+9iGcW_u9f0i00ZrVHOnOQf5bFMGzkk_dfhEesr6f8gsh=fQ@mail.gmail.com>
 <CAPTjJmp94ikiureFbkQaz81+LbDiBLpnxWGeerC+DAzGZgmkMA@mail.gmail.com>
 <nvn3k3$t94$1@blaine.gmane.org>
 <mailman.7.1478431240.30068.python-list@python.org>
 <e750549b-7e9e-4664-ae20-e989b690c920@googlegroups.com>
Message-ID: <nvnbku$3ph$1@blaine.gmane.org>

Alec Taylor wrote:

> On Sunday, November 6, 2016 at 10:20:53 PM UTC+11, Peter Otten wrote:
>> Chris Angelico wrote:
>> 
>> > On Sun, Nov 6, 2016 at 9:17 PM, Alec Taylor <alec.taylor6 at gmail.com>
>> > wrote:
>> >> Running Ubuntu 16.10 with Python 2.7.12+ (default one) and virtualenv
>> >> 15.0.3 (`sudo -H pip install virtualenv`). What am I doing wrong?
>> >>
>> >>     $ virtualenv a && . "$_"/bin/activate && pip --version
>> > 
>> > I'm pretty sure virtualenv (like venv, about which I'm certain)
>> > creates something that you have to 'source' into your shell, rather
>> > than running in the classic way:
>> > 
>> > source env/bin/activate
>> 
>> I think this is what the
>> 
>> . "$_"/bin/activate
>> 
>> part of Alec's command is supposed to do.
>> 
>> Yes, that's a dot, not grit on Tim's screen ;)
>> 
>> > It needs to alter environment variables in your shell, which can't be
>> > done from a separate program.
> 
> Hmm:
> $ echo $PYTHONPATH
> /usr/local/lib/python2.7/site-packages:/usr/local/lib/python2.7/dist-
packages
> $ pip --version
> pip 9.0.0 from /tmp/a/local/lib/python2.7/site-packages (python 2.7)
> 
> 
> So that worked. Shouldn't PYTHONPATH be set/upserted by the activation of
> the virtualenv?

That would have been my expectation, but I don't feel competent to recommend 
changes in pip. Also, to avoid mixing Python 2 and 3, I tend to use pth 
files to configure non-standard module locations.




From tjreedy at udel.edu  Sun Nov  6 08:41:41 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 6 Nov 2016 08:41:41 -0500
Subject: passing a variable to cmd
In-Reply-To: <d93c29f8-adee-4657-9f65-e1ef95208a79@googlegroups.com>
References: <d93c29f8-adee-4657-9f65-e1ef95208a79@googlegroups.com>
Message-ID: <nvnbui$qm4$1@blaine.gmane.org>

On 11/6/2016 6:48 AM, SS wrote:

> cmd="dig @4.2.2.2 %s ns +short", % (domname)
>
> does not work.

No kidding.  ', %' is a syntax error.  The , makes a tuple, the % after 
string does interpolation.  You obviously want the latter so omit the ,.

The traceback should have pointed you to where the code became invalid.

 >>> cmd="dig @4.2.2.2 %s ns +short", % (domname)
   File "<stdin>", line 1
     cmd="dig @4.2.2.2 %s ns +short", % (domname)
                                      ^
This mean that % is not valid GIVEN WHAT HAS COME BEFORE being treated 
as correct.  The , could have been correct, so Python cannot know it is 
incorrect here.

-- 
Terry Jan Reedy



From rosuav at gmail.com  Sun Nov  6 08:59:57 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 7 Nov 2016 00:59:57 +1100
Subject: passing a variable to cmd
In-Reply-To: <d93c29f8-adee-4657-9f65-e1ef95208a79@googlegroups.com>
References: <d93c29f8-adee-4657-9f65-e1ef95208a79@googlegroups.com>
Message-ID: <CAPTjJmqnRgk8LFFv-3VBDXhWHuJBUf2sj1RRqwDVi=O2WGJG_A@mail.gmail.com>

On Sun, Nov 6, 2016 at 10:48 PM, SS <sami.strat at gmail.com> wrote:
> # cmd='dig @4.2.2.2 nbc.com ns +short'
> cmd="dig @4.2.2.2 %s ns +short", % (domname)
> proc=subprocess.Popen(shlex.split(cmd),stdout=subprocess.PIPE)
> out,err=proc.communicate()
> print(out)
>
> The line that is commented out works fine.  However, I want to substitute a variable for "nbc.com".  The command:

Then don't use a single string; use a list of strings:

cmd = ["dig", "@4.2.2.2", domname, "ns", "+short"]

This is what shlex.split does, and you're fighting against things to
try to force everything through that one channel. What happens if
someone puts a space in domname? It should come back with an error
from dig (you can't have spaces in domain names!), but with naive
string interpolation, you would be passing multiple separate
arguments.

Get used to using lists of arguments for all command execution. Python
isn't optimized for keyboard shorthands in running subcommands, the
way an interactive shell is; it's designed more to be reliable and
dependable. I wouldn't accept a shell that forced me to type commands
with all their parts quoted, but I also don't use single strings with
Python very often. This is the safer way.

ChrisA


From rosuav at gmail.com  Sun Nov  6 09:06:52 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 7 Nov 2016 01:06:52 +1100
Subject: Force virtualenv pip to be used
In-Reply-To: <nvn3k3$t94$1@blaine.gmane.org>
References: <CAO+9iGcW_u9f0i00ZrVHOnOQf5bFMGzkk_dfhEesr6f8gsh=fQ@mail.gmail.com>
 <CAPTjJmp94ikiureFbkQaz81+LbDiBLpnxWGeerC+DAzGZgmkMA@mail.gmail.com>
 <nvn3k3$t94$1@blaine.gmane.org>
Message-ID: <CAPTjJmoahPATQ2oMSwAdzVhKkWLv_yYR9HXcLF-d8X0o0XeTBw@mail.gmail.com>

On Sun, Nov 6, 2016 at 10:19 PM, Peter Otten <__peter__ at web.de> wrote:
> Chris Angelico wrote:
>
>> On Sun, Nov 6, 2016 at 9:17 PM, Alec Taylor <alec.taylor6 at gmail.com>
>> wrote:
>>> Running Ubuntu 16.10 with Python 2.7.12+ (default one) and virtualenv
>>> 15.0.3 (`sudo -H pip install virtualenv`). What am I doing wrong?
>>>
>>>     $ virtualenv a && . "$_"/bin/activate && pip --version
>>
>> I'm pretty sure virtualenv (like venv, about which I'm certain)
>> creates something that you have to 'source' into your shell, rather
>> than running in the classic way:
>>
>> source env/bin/activate
>
> I think this is what the
>
> . "$_"/bin/activate
>
> part of Alec's command is supposed to do.
>
> Yes, that's a dot, not grit on Tim's screen ;)

Yep, I see that now. Guess my screen's dirty again. Sorry!

There are a few possibilities still.

1) You *are* running all this from /tmp, right? "virtualenv a" creates
a subdirectory off the current directory, and then you look for
/tmp/a.

2) Is there an esoteric interaction between the bash "&&" and the
source command?

3) virtualenv could behave differently from venv. It's a third-party
package that works by hacks, compared to the properly-integrated venv
module.

Further research is required.

ChrisA


From torriem at gmail.com  Sun Nov  6 09:35:19 2016
From: torriem at gmail.com (Michael Torrie)
Date: Sun, 6 Nov 2016 07:35:19 -0700
Subject: pip3 : command not found
In-Reply-To: <slrno1tfqa.sd6.jon+usenet@sable.unequivocal.eu>
References: <CAFpV4RpSbFLcBG4=Ax9jXzKEzD1Dg9N+bkST3VG-3NUtHS9Y4g@mail.gmail.com>
 <85zill9wv5.fsf@benfinney.id.au>
 <mailman.13.1477874313.17234.python-list@python.org>
 <slrno1dvok.2d7.jon+usenet@sable.unequivocal.eu>
 <5817231d$0$1593$c3e8da3$5496439d@news.astraweb.com>
 <slrno1rai3.sd6.jon+usenet@sable.unequivocal.eu>
 <581dc12e$0$1617$c3e8da3$5496439d@news.astraweb.com>
 <slrno1s095.sd6.jon+usenet@sable.unequivocal.eu>
 <581e0e81$0$1598$c3e8da3$5496439d@news.astraweb.com>
 <slrno1shqa.sd6.jon+usenet@sable.unequivocal.eu>
 <581e8817$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <slrno1tato.sd6.jon+usenet@sable.unequivocal.eu>
 <CAPTjJmqHuYGhC77_mPzztqv8b1_gm_1LZDfPn3StSvJfo0CDag@mail.gmail.com>
 <mailman.0.1478406205.30068.python-list@python.org>
 <slrno1tfqa.sd6.jon+usenet@sable.unequivocal.eu>
Message-ID: <1fbe3010-6ec1-8173-3154-ccca2d50b60e@gmail.com>

On 11/05/2016 11:27 PM, Jon Ribbens wrote:
>> 2) If Python notices that its executable comes from a venv, it uses it.
> 
> Yes. My question is *how does it notice*?

I'm guessing that it notices by examining the path it was launched from
and looks for virtual environment files relative to that path.

Here's what a random site said about this (I didn't search long enough
to find the official documentation on python.org):

"When Python is starting up, it looks at the path of its binary (which,
in a virtual environment, is actually just a copy of, or symlink to,
your system?s Python binary). It then sets the location of sys.prefix
and sys.exec_prefix based on this location, omitting the ?bin? portion
of the path."

This is actually a common idiom in the unix world.  For example, busybox
works on this principle. You link the busybox executable and call it,
say, "cat" and when involved, busybox looks to see how what path it was
launched via and then behaves accordingly.



From rosuav at gmail.com  Sun Nov  6 11:12:25 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 7 Nov 2016 03:12:25 +1100
Subject: Pre-pep discussion material: in-place equivalents to map and
 filter
In-Reply-To: <CAPpswAJebKkALVmmzDd45fOU_G34QGUAJfuZ8exeDPcZSGfC3g@mail.gmail.com>
References: <8a268e42-87c4-4054-bd19-1dc417dd6d01@googlegroups.com>
 <581af57a$0$1585$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmrQdo2J2ztRz-waim2rsaugi-CJDzOrFe441-OqgoS9Pg@mail.gmail.com>
 <mailman.17.1478181910.7975.python-list@python.org>
 <581b6d3a$0$1583$c3e8da3$5496439d@news.astraweb.com>
 <CAPpswAJF7PzVb1nyqD0fj=2SuW7pT6QNrH_vz1+CVSNGU-Pv2Q@mail.gmail.com>
 <mailman.26.1478204949.7975.python-list@python.org>
 <581d9b90$0$1599$c3e8da3$5496439d@news.astraweb.com>
 <CAPpswAJQQQAr2eZayvARMZf--BFM_D7PWoitzbmvvqfAQgzB0w@mail.gmail.com>
 <CAPTjJmoyg=QgUbjUvb8EsvL4sS35NAp8VXOfQ1xNxyE_wNqBgQ@mail.gmail.com>
 <CAPpswAJebKkALVmmzDd45fOU_G34QGUAJfuZ8exeDPcZSGfC3g@mail.gmail.com>
Message-ID: <CAPTjJmpHahEs2xKsmeWKSeoci4=5x+MbHhJeUoNN7tPo9Yg14w@mail.gmail.com>

On Sun, Nov 6, 2016 at 12:50 AM, Arthur Havlicek
<arthurhavlicek at gmail.com> wrote:
> 2016-11-05 12:47 GMT+01:00 Chris Angelico <rosuav at gmail.com>:
>
>> On Sat, Nov 5, 2016 at 9:50 PM, Arthur Havlicek
>>
>> But here's the thing. For everyone who writes a decorator function,
>> there could be dozens who use it.
>>
>
> The day that one guy leaves the team, suddenly you have code that's become
> a bit tricky to maintain.

True, and this can become a problem when those dozens have no
comprehension of how these features work. But fortunately, all it
takes is for one person to step up and learn how decorators are
written, and the problem is solved. (And it's not that hard. We teach
decorator authorship in our Web Programming In Python course. Not in a
huge amount of detail, but enough that a student will be able to
carefully tease apart the code of a decorator function and figure out
what it's actually doing.)

ChrisA


From rosuav at gmail.com  Sun Nov  6 11:15:30 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 7 Nov 2016 03:15:30 +1100
Subject: No subject
In-Reply-To: <CA++xw78t1YmN=_WKHEX7u_GUPMOdUz6enWhR1VDh=aNG5fYvOw@mail.gmail.com>
References: <CA++xw78t1YmN=_WKHEX7u_GUPMOdUz6enWhR1VDh=aNG5fYvOw@mail.gmail.com>
Message-ID: <CAPTjJmpe80xyeTDF6+2VRHaGWpNoYwYugBgjXycjhSgHLpCGRw@mail.gmail.com>

On Sun, Nov 6, 2016 at 12:33 AM, Abdullahi Salihu Abubakar
<salihuabdu at gmail.com> wrote:
> [lots of text, taking a representative sample]
> 4.   I presently use list to hold the list of beneficiaries. Based on the
> short description of the proposed app above, which other container is it
> suitable for holding the categories of the beneficiaries?
>
> I am sorry if I omit any necessary information. This is how far I have
> gone. I am always ready to provide any necessary details please. Am a
> newbie, but always ready to learn, no matter what.
>

It sounds to me like you have some code. The best way to ask for help
about code is to show the code. Show us the smallest program that
showcases the problem you're having, and we can help you with it.

ChrisA


From g.rodola at gmail.com  Sun Nov  6 14:28:52 2016
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Sun, 6 Nov 2016 20:28:52 +0100
Subject: ANN: released psutil 5.0.0,
 introducing a 2x speedup for process methods
Message-ID: <CAFYqXL9BiTGt8Lvz8jXjdqf+ZVF9NoXLXTJd6dp5G3NROAcMFA@mail.gmail.com>

Hello all,
I'm glad to announce the release of psutil 5.0.0:
https://github.com/giampaolo/psutil
This release introduces important speedups making psutil from 2x to 6x
faster depending on what platform you're on.
A full blog post can be found here:
http://grodola.blogspot.com/2016/11/psutil-500-is-around-twice-as-fast.html

About
=====

psutil (process and system utilities) is a cross-platform library for
retrieving information on running processes and system utilization (CPU,
memory, disks, network) in Python. It is useful mainly for system
monitoring, profiling and limiting process resources and management of
running processes. It implements many functionalities offered by command
line tools such as: ps, top, lsof, netstat, ifconfig, who, df, kill, free,
nice, ionice, iostat, iotop, uptime, pidof, tty, taskset, pmap. It
currently supports Linux, Windows, OSX, Sun Solaris, FreeBSD, OpenBSD and
NetBSD, both 32-bit and 64-bit architectures, with Python versions from 2.6
to 3.5 (users of Python 2.4 and 2.5 may use 2.1.3 version). PyPy is also
known to work.

What's new
==========

*2016-11-06*

**Enhncements**

- #799: new Process.oneshot() context manager making Process
  methods around +2x faster in general and from +2x to +6x faster on
  Windows.
- #943: better error message in case of version conflict on import.

**Bug fixes**

- #932: [NetBSD] net_connections() and Process.connections() may fail
without
  raising an exception.
- #933: [Windows] memory leak in cpu_stats() and
WindowsService.description().

Links
=====

- Home page: https://github.com/giampaolo/psutil
- Download: https://pypi.python.org/pypi/psutil
- Documentation: http://pythonhosted.org/psutil
- What's new: https://github.com/giampaolo/psutil/blob/master/HISTORY.rst

--

Giampaolo - http://grodola.blogspot.com


From paulavijit928 at gmail.com  Sun Nov  6 14:36:25 2016
From: paulavijit928 at gmail.com (Avijit Paul)
Date: Mon, 7 Nov 2016 01:36:25 +0600
Subject: Fwd: About Installation.
In-Reply-To: <CANS-2mDtq6joNaZEfw78QN8CxofYYhS3YbOBn0WwKyKnEvN5iQ@mail.gmail.com>
References: <CANS-2mDtq6joNaZEfw78QN8CxofYYhS3YbOBn0WwKyKnEvN5iQ@mail.gmail.com>
Message-ID: <CANS-2mAHBE-V4w6fiQwfSuDM-5asC3VqOpfb4Jt_dqON4OVLvA@mail.gmail.com>

---------- Forwarded message ----------
From: Avijit Paul <paulavijit928 at gmail.com>
Date: Mon, Nov 7, 2016 at 1:27 AM
Subject: About Installation.
To: python-list at python.org


Hello,

I installed python-3.6.0b2-amd64 on my Windows PC. Which is running on
64-bit system of Windows 8.1. The installation was successful. After the
installation, when I ran the program it showed the following error, as
shows in the attachment.

I need your kind help to recover the error showed in the program.

Thank You.


From vano at mail.mipt.ru  Sun Nov  6 16:23:47 2016
From: vano at mail.mipt.ru (Ivan Pozdeev)
Date: Mon, 7 Nov 2016 00:23:47 +0300
Subject: What is currently the recommended way to work with a distutils-based
 setup.py that requires compilation?
Message-ID: <9da79944-6bf3-0a65-a4ab-197f5799af4d@mail.mipt.ru>

https://wiki.python.org/moin/WindowsCompilers has now completely 
replaced instructions for `distutils`-based packages (starting with 
`from distutils.core import setup`) with ones for `setuptools`-based 
ones (starting with `from setuptools import setup`).

However, if I have a `distutils`-based `setup.py`, when I run it, 
`setuptools` is not used - thus the instructions on the page don't work.

It is possible to run a `distutils`-based script through `setuptools`, 
as `pip` does, but it requires the following code 
(https://github.com/pypa/pip/blob/8.1.2/pip/req/req_install.py#L849 ):

     python -u -c "import setuptools, tokenize;__file__=<setup.py full 
path>;
         exec(compile(getattr(tokenize, 'open', open)(__file__).read()
         .replace('\\r\\n', '\\n'), __file__, 'exec'))" <setup.py arguments>

They can't possibly expect me to type that on the command line each 
time, now can they?

I also asked this at http://stackoverflow.com/q/40174932/648265 a couple 
of days ago (to no avail).

-- 

Regards,
Ivan



From rosefox911 at gmail.com  Sun Nov  6 18:24:02 2016
From: rosefox911 at gmail.com (rosefox911 at gmail.com)
Date: Sun, 6 Nov 2016 15:24:02 -0800 (PST)
Subject: Delete h2 until you reach the next h2 in beautifulsoup
In-Reply-To: <c9c99de5-3b80-45b4-8e14-509fef08f3d5@googlegroups.com>
References: <c9c99de5-3b80-45b4-8e14-509fef08f3d5@googlegroups.com>
Message-ID: <bc6bb3b6-53cd-4d3e-ac6e-f12700b8ad9e@googlegroups.com>

On Sunday, November 6, 2016 at 1:27:48 AM UTC-4, rosef... at gmail.com wrote:
> Considering the following html:
> 
>     <h2 id="example">cool stuff</h2> <ul> <li>hi</li> </ul> <div> <h2 id="cool"><h2> <ul><li>zz</li> </ul> </div>
> 
> and the following list:
> 
>     ignore_list = ['example','lalala']
> 
> My goal is, while going through the HTML using Beautifulsoup, I find a h2 that has an ID that is in my list (ignore_list) I should delete all the ul and lis under it until I find another h2. I would then check if the next h2 was in my ignore list, if it is, delete all the ul and lis until I reach the next h2 (or if there are no h2s left, delete the ul and lis under the current one and stop). 
> 
> How I see the process going: you read all the h2s from up to down in the DOM. If the id for any of those is in the ignore_list, then delete all the ul and li under the h2 until you reach the NEXT h2. If there is no h2, then delete the ul and LI then stop.
> 
> Here is the full HMTL I am trying to work with: http://pastebin.com/Z3ev9c8N
> 
> I am trying to delete all the UL and lis after "See_also"How would I accomplish this in Python?


I got it working with the following solution:

#Remove content I don't want
            try:
                for element in body.find_all('h2'):
                    current_h2 = element.get_text()
                    current_h2 = current_h2.replace('[edit]','')
                    #print(current_h2)
                    if(current_h2 in ignore_list):
                        if(element.find_next_sibling('div') != None):
                            element.find_next_sibling('div').decompose()
                        if(element.find_next_sibling('ul') != None):
                            element.find_next_sibling('ul').decompose()
            except(AttributeError, TypeError) as e:
                continue   


From pavel.aronsky at gmail.com  Sun Nov  6 20:11:45 2016
From: pavel.aronsky at gmail.com (ddbug)
Date: Sun, 6 Nov 2016 17:11:45 -0800 (PST)
Subject: Confused with installing per-user in Windows
Message-ID: <79b0d790-5d2b-4c45-9ba2-30bd581f1bab@googlegroups.com>

Dear experts,

I need to install some scripts for current user (to skip sudo, UAC popups and whatever).

So I make a sdist and use python -m pip install --user ....

This should work for either Python 2 or 3.

On Linux, pip installs the scripts into ~/.local/bin ; users are instructed to add this to their PATH if they have not done so already.

In Windows, the user-local directory for scripts is %APPDATA%\Python\Scripts. It is not in PATH by default and finding it is hard (because Microsoft made it hidden in their infinite wisdom).

But more to this, either Python (2.7 or 3.5) will NOT look there by default. When user types "python myscript.py" or "py myscript.py" he is baffled by "not found".

Now, the question: 

1. would it be good if python interpreter could JUST find user-local scripts - by default or by some easy configuration option?

2. If not, would it be good to put this smartness into the PY.EXE launcher, make this behavior default or by a simple command line option?

So that user can be instructed to type "py myscript [.py]"  and it will JUST work, if the script is on existing PATH or in the per-user directory?


I know about bdist_wininst and Windows specific install options, but prefer single sdist installer whenever possible.

Thanks for reading.

--d


From pavel.aronsky at gmail.com  Sun Nov  6 20:19:21 2016
From: pavel.aronsky at gmail.com (ddbug)
Date: Sun, 6 Nov 2016 17:19:21 -0800 (PST)
Subject: Confused with installing per-user in Windows
In-Reply-To: <79b0d790-5d2b-4c45-9ba2-30bd581f1bab@googlegroups.com>
References: <79b0d790-5d2b-4c45-9ba2-30bd581f1bab@googlegroups.com>
Message-ID: <5cd37595-5029-44a2-aaf5-22214d28a8e6@googlegroups.com>

So basically I want to modify py.exe to not only detect the Python version from a script file, but also help locating the script file.

-- d


From vano at mail.mipt.ru  Sun Nov  6 20:39:20 2016
From: vano at mail.mipt.ru (Ivan Pozdeev)
Date: Mon, 7 Nov 2016 04:39:20 +0300
Subject: Confused with installing per-user in Windows
In-Reply-To: <79b0d790-5d2b-4c45-9ba2-30bd581f1bab@googlegroups.com>
References: <79b0d790-5d2b-4c45-9ba2-30bd581f1bab@googlegroups.com>
Message-ID: <614e2fb2-762a-dc64-632f-e01f558b8468@mail.mipt.ru>

On 07.11.2016 4:11, ddbug wrote:
> Dear experts,
>
> I need to install some scripts for current user (to skip sudo, UAC popups and whatever).
>
> So I make a sdist and use python -m pip install --user ....
>
> This should work for either Python 2 or 3.
>
> On Linux, pip installs the scripts into ~/.local/bin ; users are instructed to add this to their PATH if they have not done so already.
>
> In Windows, the user-local directory for scripts is %APPDATA%\Python\Scripts. It is not in PATH by default and finding it is hard (because Microsoft made it hidden in their infinite wisdom).
>
> But more to this, either Python (2.7 or 3.5) will NOT look there by default. When user types "python myscript.py" or "py myscript.py" he is baffled by "not found".
If `myscript.py' is in the system-wide Scripts (or anywhere else other 
that current dir), "python myscript.py" will yield "not found" all the same.
>
> Now, the question:
>
> 1. would it be good if python interpreter could JUST find user-local scripts - by default or by some easy configuration option?
Just append `%APPDATA%\Python\Scripts' into your user-specific PATH and 
use "myscript.py" (without "python") - that will search for the file on 
PATH and run it with whatever program you have associated with the `.py' 
extension (which should be `py.exe' if you're using it).
>
> 2. If not, would it be good to put this smartness into the PY.EXE launcher, make this behavior default or by a simple command line option?
>
> So that user can be instructed to type "py myscript [.py]"  and it will JUST work, if the script is on existing PATH or in the per-user directory?
>
>
> I know about bdist_wininst and Windows specific install options, but prefer single sdist installer whenever possible.
>
> Thanks for reading.
>
> --d

-- 
Regards,
Ivan



From python at mrabarnett.plus.com  Sun Nov  6 21:35:42 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Mon, 7 Nov 2016 02:35:42 +0000
Subject: Fwd: About Installation.
In-Reply-To: <CANS-2mAHBE-V4w6fiQwfSuDM-5asC3VqOpfb4Jt_dqON4OVLvA@mail.gmail.com>
References: <CANS-2mDtq6joNaZEfw78QN8CxofYYhS3YbOBn0WwKyKnEvN5iQ@mail.gmail.com>
 <CANS-2mAHBE-V4w6fiQwfSuDM-5asC3VqOpfb4Jt_dqON4OVLvA@mail.gmail.com>
Message-ID: <080aec8a-188d-2777-e182-e5ea816b2ab1@mrabarnett.plus.com>

On 2016-11-06 19:36, Avijit Paul wrote:
>
> I installed python-3.6.0b2-amd64 on my Windows PC. Which is running on
> 64-bit system of Windows 8.1. The installation was successful. After the
> installation, when I ran the program it showed the following error, as
> shows in the attachment.
>
> I need your kind help to recover the error showed in the program.
>
This list doesn't support attachments, so I'm guessing that it's 
complaining that it can't find "api-ms-win-crd-runtime-l1-1-0.dll".

You should already have it if you've kept your PC up to date via Windows 
Update, but if you haven't, read this:

https://support.microsoft.com/en-us/kb/3118401



From jon+usenet at unequivocal.eu  Mon Nov  7 00:11:48 2016
From: jon+usenet at unequivocal.eu (Jon Ribbens)
Date: Mon, 7 Nov 2016 05:11:48 -0000 (UTC)
Subject: pip3 : command not found
References: <CAFpV4RpSbFLcBG4=Ax9jXzKEzD1Dg9N+bkST3VG-3NUtHS9Y4g@mail.gmail.com>
 <85zill9wv5.fsf@benfinney.id.au>
 <mailman.13.1477874313.17234.python-list@python.org>
 <slrno1dvok.2d7.jon+usenet@sable.unequivocal.eu>
 <5817231d$0$1593$c3e8da3$5496439d@news.astraweb.com>
 <slrno1rai3.sd6.jon+usenet@sable.unequivocal.eu>
 <581dc12e$0$1617$c3e8da3$5496439d@news.astraweb.com>
 <slrno1s095.sd6.jon+usenet@sable.unequivocal.eu>
 <581e0e81$0$1598$c3e8da3$5496439d@news.astraweb.com>
 <slrno1shqa.sd6.jon+usenet@sable.unequivocal.eu>
 <581e8817$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <slrno1tato.sd6.jon+usenet@sable.unequivocal.eu>
 <CAPTjJmqHuYGhC77_mPzztqv8b1_gm_1LZDfPn3StSvJfo0CDag@mail.gmail.com>
 <mailman.0.1478406205.30068.python-list@python.org>
 <slrno1tfqa.sd6.jon+usenet@sable.unequivocal.eu>
 <CAPTjJmrs+LtPh44o43xM8v4rjgH9MXneQ-zSuj9Y1iusvvqCRQ@mail.gmail.com>
 <mailman.6.1478430795.30068.python-list@python.org>
Message-ID: <slrno2039h.sd6.jon+usenet@sable.unequivocal.eu>

On 2016-11-06, Chris Angelico <rosuav at gmail.com> wrote:
> On Sun, Nov 6, 2016 at 4:27 PM, Jon Ribbens <jon+usenet at unequivocal.eu> wrote:
>>> 2) If Python notices that its executable comes from a venv, it uses it.
>>
>> Yes. My question is *how does it notice*?
>
> I could answer this question, but since you don't appear to be
> following the thread properly, I'll point out that it's already been
> said up above. Go read it.

That's not particularly helpful. I have read the entire thread and
the question does not appear to have been answered. Could you at
least provide the date/date/from (if not a Message-ID) of the post
you are talking about please?


From jon+usenet at unequivocal.eu  Mon Nov  7 00:59:43 2016
From: jon+usenet at unequivocal.eu (Jon Ribbens)
Date: Mon, 7 Nov 2016 05:59:43 -0000 (UTC)
Subject: pip3 : command not found
References: <CAFpV4RpSbFLcBG4=Ax9jXzKEzD1Dg9N+bkST3VG-3NUtHS9Y4g@mail.gmail.com>
 <85zill9wv5.fsf@benfinney.id.au>
 <mailman.13.1477874313.17234.python-list@python.org>
 <slrno1dvok.2d7.jon+usenet@sable.unequivocal.eu>
 <5817231d$0$1593$c3e8da3$5496439d@news.astraweb.com>
 <slrno1rai3.sd6.jon+usenet@sable.unequivocal.eu>
 <581dc12e$0$1617$c3e8da3$5496439d@news.astraweb.com>
 <slrno1s095.sd6.jon+usenet@sable.unequivocal.eu>
 <581e0e81$0$1598$c3e8da3$5496439d@news.astraweb.com>
 <slrno1shqa.sd6.jon+usenet@sable.unequivocal.eu>
 <581e8817$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <slrno1tato.sd6.jon+usenet@sable.unequivocal.eu>
 <CAPTjJmqHuYGhC77_mPzztqv8b1_gm_1LZDfPn3StSvJfo0CDag@mail.gmail.com>
 <mailman.0.1478406205.30068.python-list@python.org>
 <slrno1tfqa.sd6.jon+usenet@sable.unequivocal.eu>
 <1fbe3010-6ec1-8173-3154-ccca2d50b60e@gmail.com>
 <mailman.13.1478442929.30068.python-list@python.org>
Message-ID: <slrno2063b.sd6.jon+usenet@sable.unequivocal.eu>

On 2016-11-06, Michael Torrie <torriem at gmail.com> wrote:
> I'm guessing that it notices by examining the path it was launched from
> and looks for virtual environment files relative to that path.

Indeed, the mysterious thing is what are "virtual environment files"?
The official docs say "A venv is a directory tree which contains
Python executable files and other files which indicate that it is a
venv." but don't explain what they mean by "other files".

> Here's what a random site said about this (I didn't search long enough
> to find the official documentation on python.org):
>
> "When Python is starting up, it looks at the path of its binary (which,
> in a virtual environment, is actually just a copy of, or symlink to,
> your system?s Python binary). It then sets the location of sys.prefix
> and sys.exec_prefix based on this location, omitting the ?bin? portion
> of the path."

Aha! This is progress. OK, I've done some more investigation and it
looks like the above description is not exactly accurate.

The magic is actually all in site.py by the looks of it. This looks
at argv[0] (effectively - it's actually using sys.executable) and
then looks for ./pyvenv.cfg and ../pyvenv.cfg relative to that file.
Only if one of those files exists does it then set sys.prefix to ..
relative to sys.executable and then sys.path is set using various
hard-coded paths relative to sys.prefix.

So the answer to my original question is that Python knows it's in a
venv "if there exists a file called pyvenv.cfg in the same directory
as or the parent directory of the python exeuctable".

It also appears that this feature was added in Python 3.3, so being
able to run the venv python directly presumably only works on that
version and later.


From dieter at handshake.de  Mon Nov  7 02:24:36 2016
From: dieter at handshake.de (dieter)
Date: Mon, 07 Nov 2016 08:24:36 +0100
Subject: [Theory] How to speed up python code execution / pypy vs GPU
References: <nvl3q7$vut$1@news.icm.edu.pl>
Message-ID: <87inrz4urv.fsf@handshake.de>

"Mr. Wrobel" <mr at e-wrobel.pl> writes:
> ...
> However the same skeptics told my that, ok we believe that it is true,
> however the code execution is much slower than any other compiled
> language.

However, in many cases "code execution speed" is not the primary concern.
In my experience, "development speed" is far superior with Python
than with e.g. C, C++ or Java.

Even for programs for which "code execution speed" is important, the importance
often affects only quite local portions of the program. For those portions,
I look for existing C/C++ libraries or I implement them myself
in "C/C++", maybe with the help of "cython".



From alec.taylor6 at gmail.com  Mon Nov  7 03:02:50 2016
From: alec.taylor6 at gmail.com (Alec Taylor)
Date: Mon, 7 Nov 2016 00:02:50 -0800 (PST)
Subject: Force virtualenv pip to be used
In-Reply-To: <mailman.12.1478441219.30068.python-list@python.org>
References: <CAO+9iGcW_u9f0i00ZrVHOnOQf5bFMGzkk_dfhEesr6f8gsh=fQ@mail.gmail.com>
 <CAPTjJmp94ikiureFbkQaz81+LbDiBLpnxWGeerC+DAzGZgmkMA@mail.gmail.com>
 <nvn3k3$t94$1@blaine.gmane.org>
 <CAPTjJmoahPATQ2oMSwAdzVhKkWLv_yYR9HXcLF-d8X0o0XeTBw@mail.gmail.com>
 <mailman.12.1478441219.30068.python-list@python.org>
Message-ID: <8c96676e-56a2-4b6a-be13-ff6c4a000649@googlegroups.com>

On Monday, November 7, 2016 at 1:07:12 AM UTC+11, Chris Angelico wrote:
> On Sun, Nov 6, 2016 at 10:19 PM, Peter Otten <__peter__ at web.de> wrote:
> > Chris Angelico wrote:
> >
> >> On Sun, Nov 6, 2016 at 9:17 PM, Alec Taylor <alec.taylor6 at gmail.com>
> >> wrote:
> >>> Running Ubuntu 16.10 with Python 2.7.12+ (default one) and virtualenv
> >>> 15.0.3 (`sudo -H pip install virtualenv`). What am I doing wrong?
> >>>
> >>>     $ virtualenv a && . "$_"/bin/activate && pip --version
> >>
> >> I'm pretty sure virtualenv (like venv, about which I'm certain)
> >> creates something that you have to 'source' into your shell, rather
> >> than running in the classic way:
> >>
> >> source env/bin/activate
> >
> > I think this is what the
> >
> > . "$_"/bin/activate
> >
> > part of Alec's command is supposed to do.
> >
> > Yes, that's a dot, not grit on Tim's screen ;)
> 
> Yep, I see that now. Guess my screen's dirty again. Sorry!
> 
> There are a few possibilities still.
> 
> 1) You *are* running all this from /tmp, right? "virtualenv a" creates
> a subdirectory off the current directory, and then you look for
> /tmp/a.
> 
> 2) Is there an esoteric interaction between the bash "&&" and the
> source command?
> 
> 3) virtualenv could behave differently from venv. It's a third-party
> package that works by hacks, compared to the properly-integrated venv
> module.
> 
> Further research is required.
> 
> ChrisA

venv seems to be a Python 3 thing. I'm using Python 2. Will probably experiment with Python 3 also and get cross compatibility, but then I'd still use virtualenv so I get consistent tooling.

And the issue has been found: PYTHONPATH being set caused the problem.


From eryksun at gmail.com  Mon Nov  7 03:09:08 2016
From: eryksun at gmail.com (eryk sun)
Date: Mon, 7 Nov 2016 08:09:08 +0000
Subject: Confused with installing per-user in Windows
In-Reply-To: <79b0d790-5d2b-4c45-9ba2-30bd581f1bab@googlegroups.com>
References: <79b0d790-5d2b-4c45-9ba2-30bd581f1bab@googlegroups.com>
Message-ID: <CACL+1avbkaXfnuupsLTjHHtc1=+nmU-_1_w=ejaA9yGx+qYhnA@mail.gmail.com>

On Mon, Nov 7, 2016 at 1:11 AM, ddbug <pavel.aronsky at gmail.com> wrote:
>
> In Windows, the user-local directory for scripts is %APPDATA%\Python\Scripts. It is not in
> PATH by default and finding it is hard (because Microsoft made it hidden in their infinite
> wisdom).

POSIX "~/.local" is hidden as well, by convention, so I don't see how
the directory's being hidden is relevant.

In Windows 10, showing hidden files and folders in Explorer takes just
two clicks -- three if the ribbon is collapsed. It's not much harder
in Windows 7, but you have to know how to set folder options. Or enter
%appdata% in the location bar, or shell:appdata:

    http://www.winhelponline.com/blog/shell-commands-to-access-the-special-folders

In cmd use "dir /a" to list all files or "dir /ad" to list all
directories. You can use set "DIRCMD=/a" if you prefer to always list
all files. (FYI, /a doesn't mean [a]ll; it's an [a]ttribute filter; an
empty filter yields all files and directories.)

In PowerShell it's "gci -fo" or "gci -fo -ad". To be verbose, the
latter is "Get-ChildItem -Force -Attributes Directory".

I do think using %APPDATA% is a mistake, but not because it's hidden.
A --user install should be using %LOCALAPPDATA%, which is excluded
from the user's roaming profile. It was also a mistake to dump
everything into a common "Scripts" directory, since the Python version
number isn't appended to script names. This was fixed in 3.5, which
instead uses "%APPDATA%\Python\Python35\Scripts".

> 1. would it be good if python interpreter could JUST find user-local scripts - by default or by some
> easy configuration option?

That would be surprising behavior if it were enabled by default.

It was already suggested that you add the user scripts directory to
PATH and run the .py file directly. However, calling ShellExecuteEx to
run a file by association isn't always an option. That's why pip and
setuptools create EXE wrappers for script entry points when installing
source and wheel distributions.

Consider packaging your scripts in a wheel that defines entry points.
For example:

spam_scripts\spam.py:

    import sys

    def main():
        print('spam%d%d' % sys.version_info[:2])
        return 42

    # unused classic entry point
    if __name__ == '__main__':
        sys.exit(main())

setup.py:

    from setuptools import setup

    setup(name='spam_scripts',
          version='1.0',
          description='...',
          url='...',
          author='...',
          author_email='...',
          license='...',
          packages=['spam_scripts'],
          entry_points = {
              'console_scripts': ['spam=spam_scripts.spam:main'],
          })

build:

    > python setup.py bdist_wheel --universal
    > cd dist

installation (3.5.2 on this system):

    > pip install --user spam_scripts-1.0-py2.py3-none-any.whl

installation (latest 2.x):

    > py -2 -m pip install --user spam_scripts-1.0-py2.py3-none-any.whl

Usage:

    >%APPDATA%\Python\Python35\Scripts\spam.exe
    spam35
    >echo %errorlevel%
    42

    >%APPDATA%\Python\Scripts\spam.exe
    spam27
    >echo %errorlevel%
    42


From teppo.pera at gmail.com  Mon Nov  7 03:46:25 2016
From: teppo.pera at gmail.com (teppo.pera at gmail.com)
Date: Mon, 7 Nov 2016 00:46:25 -0800 (PST)
Subject: constructor classmethods
In-Reply-To: <mailman.18.1478184334.7975.python-list@python.org>
References: <7f7156ed-e396-460e-8eb1-fb6b17bab069@googlegroups.com>
 <02dc88d9-0fd3-416b-8aac-eb2b530d0320@googlegroups.com>
 <581B4D88.9010803@stoneleaf.us>
 <mailman.18.1478184334.7975.python-list@python.org>
Message-ID: <0dd29885-a328-4800-88f5-51f46cfa469d@googlegroups.com>

torstai 3. marraskuuta 2016 14.45.49 UTC Ethan Furman kirjoitti:
> On 11/03/2016 01:50 AM, teppo wrote:
> 
> > The guide is written in c++ in mind, yet the concepts stands for any
> >  programming language really. Read it through and think about it. If
> >  you come back to this topic and say: "yeah, but it's c++", then you
> >  haven't understood it.
> 
> The ideas (loose coupling, easy testing) are certainly applicable in Python -- the specific methods talked about in that paper, however, are not.

Please elaborate. Which ones explicitly?

> 
> To go back to the original example:
> 
> def __init__(self, ...):
>      self.queue = Queue()
> 
> we have several different (easy!) ways to do dependency injection:
> 
> * inject a mock Queue into the module
> * make queue a default parameter
> 
> If it's just testing, go with the first option:
> 
> import the_module_to_test
> the_module_to_test.Queue = MockQueue
> 
> and away you go.

This is doable, but how would you inject queue (if we use Queue as an example) with different variations, such as full, empty, half-full, half-empty. :) For different tests. 

> 
> If the class in question has legitimate, non-testing, reasons to specify different Queues, then make it a default argument instead:
> 
> def __init__(self, ..., queue=None):
>      if queue is None:
>          queue = Queue()
>      self.queue = queue

I already stated that this is is fine, as long as the number of arguments stays in manageable levels. Although I do think testing is good enough reason to have it injected anytime. For consistency, it makes sense to have same way to create all objects. I wouldn't suggested of using that mechanism in public API's, just in internal components.

> 
> or, if it's just for testing but you don't want to hassle injecting a MockQueue into the module itself:
> 
> def __init__(self, ..., _queue=None):
>      if _queue is None:
>          _queue = Queue()
>      self.queue = _queue
> 
> or, if the queue is only initialized (and not used) during __init__ (so you can replace it after construction with no worries):
> 
> class Example:
>      def __init__(self, ...):
>          self.queue = Queue()
> 
> ex = Example()
> ex.queue = MockQueue()
> # proceed with test

This I wouldn't recommend. It generates useless work when things start to change, especially in large code bases. Don't touch internal stuff of class in tests (or anywhere else).

> 
> The thing each of those possibilities have in common is that the normal use-case of just creating the thing and moving on is the very simple:
> 
> my_obj = Example(...)
> 
> To sum up:  your concerns are valid, but using c++ (and many other language) idioms in Python does not make good Python code.

This is not necessarily simply a c++ idiom, but a factory method design pattern 4nasm4I7%2D0s4XSH[8cS}12M)320?IqGu7_7JS$d0k+V0Dqb7. Although there are many other benefits the current pattern would offer, in this case it is primarily used just for convenience (although it seems it is not seen as such) and giving more flexibility in writing tests.

Br,
Teppo


From as at sci.fi  Mon Nov  7 03:49:54 2016
From: as at sci.fi (Anssi Saari)
Date: Mon, 07 Nov 2016 10:49:54 +0200
Subject: [PyQT] After MessageBox app quits...why?
References: <4111aac5-a0c0-928d-bfa4-7d93c6523d85@gmail.com>
 <mailman.19.1477587403.31204.python-list@python.org>
Message-ID: <vg3k2cffzd9.fsf@coffee.modeemi.fi>

Demosthenes Koptsis <demosthenesk at gmail.com> writes:

> Hello, i have a PyQT systray app with a menu and two actions.
>
> Action1 is Exit and action2 display a MessageBox with Hello World message.
>
> When i click OK to MessageBox app quits...why?
>
> http://pastebin.com/bVA49k1C

I haven't done anything with Qt in a while but apparently you need to
call QtGui.QApplication.setQuitOnLastWindowClosed(False) before
trayIcon.show().


From teppo.pera at gmail.com  Mon Nov  7 04:14:01 2016
From: teppo.pera at gmail.com (teppo.pera at gmail.com)
Date: Mon, 7 Nov 2016 01:14:01 -0800 (PST)
Subject: constructor classmethods
In-Reply-To: <mailman.19.1478184425.7975.python-list@python.org>
References: <7f7156ed-e396-460e-8eb1-fb6b17bab069@googlegroups.com>
 <02dc88d9-0fd3-416b-8aac-eb2b530d0320@googlegroups.com>
 <CAPTjJmr3c9A3ifpp-Y-5js2v7gY+Bo7us5YX89L-bTu-8sMVUw@mail.gmail.com>
 <mailman.19.1478184425.7975.python-list@python.org>
Message-ID: <fb1ead72-2041-4ea8-9b29-05e099e271e8@googlegroups.com>

torstai 3. marraskuuta 2016 14.47.18 UTC Chris Angelico kirjoitti:
> On Thu, Nov 3, 2016 at 7:50 PM,  <teppo... at gmail.com> wrote:
> > Little bit background related to this topic. It all starts from this article:
> > http://misko.hevery.com/attachments/Guide-Writing%20Testable%20Code.pdf
> >
> > The guide is written in c++ in mind, yet the concepts stands for any programming language really. Read it through and think about it. If you come back to this topic and say: "yeah, but it's c++", then you haven't understood it.
> 
> I don't have a problem with something written for C++ (though I do
> have a problem with a thirty-eight page document on how to make your
> code testable - TLDR), but do bear in mind that a *lot* of C++ code
> can be simplified when it's brought to Python. 
I know differences of c++ and python, as I have done programming in both languages for years. By the way, generally all design patterns, not just this one, are aimed to improve maintenance work, testability or flexibility and there are tons of books around these topics. This is good place to look at too:  https://sourcemaking.com/


> One Python feature that
> C++ doesn't have, mentioned already in this thread, is the way you can
> have a ton of parameters with defaults, and you then specify only
> those you want, as keyword args:
> 
> def __init__(self, important_arg1, important_arg2,
>         queue=None, cache_size=50, whatever=...):
>     pass
> 
> MyClass("foo", 123, cache_size=75)
> 
> I can ignore all the arguments that don't matter, and provide only the
> one or two that I actually need to change. Cognitive load is
> drastically reduced, compared to the "alternative constructor"
> pattern, where I have to remember not to construct anything in the
> normal way.
If writing two constructors start to feel cumbersome, it's always possible to make metaclass or decorator do at least basic one for you. For example, decorator can do __init__ to you and with that same information many other useful magic functions.

I see this more like of a trade-of of writing one factory method to help me write more stable and comprehensive tests vs having to write more brittle tests whose maintenance gets burden to developers (which can come real pain if things are done poorly in the beginning and the project goes forward). Python makes things much easier, but still care needs to be taken.

Finally, some wise words to think about (these are all tied up together):
"As an engineer, you should constantly work to make your feedback loops shorter in time and/or wider in scope."???@KentBeck

Br,
Teppo


From demosthenesk at gmail.com  Mon Nov  7 04:44:20 2016
From: demosthenesk at gmail.com (Demosthenes Koptsis)
Date: Mon, 7 Nov 2016 11:44:20 +0200
Subject: [PyQT] After MessageBox app quits...why?
In-Reply-To: <vg3k2cffzd9.fsf@coffee.modeemi.fi>
References: <4111aac5-a0c0-928d-bfa4-7d93c6523d85@gmail.com>
 <mailman.19.1477587403.31204.python-list@python.org>
 <vg3k2cffzd9.fsf@coffee.modeemi.fi>
Message-ID: <fdaafff8-17b3-3f70-f356-eb21c679f856@gmail.com>

because there is no window open and quits by default.

You have to do two things:

1) Make sure that your SystemTray class has parent a QWidget

     w = QtGui.QWidget()
     trayIcon = SystemTrayIcon(QtGui.QIcon("virtualdvd.png"), w)

2) set quit to false

     app.setQuitOnLastWindowClosed(False)

----------------------------

Example

----------------------------

def main():
     app = QtGui.QApplication(sys.argv)
     app.setQuitOnLastWindowClosed(False)

     w = QtGui.QWidget()
     trayIcon = SystemTrayIcon(QtGui.QIcon("virtualdvd.png"), w)

     trayIcon.show()
     sys.exit(app.exec_())

if __name__ == '__main__':
     main()

On 11/07/2016 10:49 AM, Anssi Saari wrote:
> Demosthenes Koptsis <demosthenesk at gmail.com> writes:
>
>> Hello, i have a PyQT systray app with a menu and two actions.
>>
>> Action1 is Exit and action2 display a MessageBox with Hello World message.
>>
>> When i click OK to MessageBox app quits...why?
>>
>> http://pastebin.com/bVA49k1C
> I haven't done anything with Qt in a while but apparently you need to
> call QtGui.QApplication.setQuitOnLastWindowClosed(False) before
> trayIcon.show().



From demosthenesk at gmail.com  Mon Nov  7 05:10:31 2016
From: demosthenesk at gmail.com (Demosthenes Koptsis)
Date: Mon, 7 Nov 2016 12:10:31 +0200
Subject: [PyQT] After MessageBox app quits...why?
In-Reply-To: <fdaafff8-17b3-3f70-f356-eb21c679f856@gmail.com>
References: <4111aac5-a0c0-928d-bfa4-7d93c6523d85@gmail.com>
 <mailman.19.1477587403.31204.python-list@python.org>
 <vg3k2cffzd9.fsf@coffee.modeemi.fi>
 <fdaafff8-17b3-3f70-f356-eb21c679f856@gmail.com>
Message-ID: <a2db093a-0063-5216-3580-b268d3b28604@gmail.com>

i answered my own question.....


On 11/07/2016 11:44 AM, Demosthenes Koptsis wrote:
> because there is no window open and quits by default.
>
> You have to do two things:
>
> 1) Make sure that your SystemTray class has parent a QWidget
>
>     w = QtGui.QWidget()
>     trayIcon = SystemTrayIcon(QtGui.QIcon("virtualdvd.png"), w)
>
> 2) set quit to false
>
>     app.setQuitOnLastWindowClosed(False)
>
> ----------------------------
>
> Example
>
> ----------------------------
>
> def main():
>     app = QtGui.QApplication(sys.argv)
>     app.setQuitOnLastWindowClosed(False)
>
>     w = QtGui.QWidget()
>     trayIcon = SystemTrayIcon(QtGui.QIcon("virtualdvd.png"), w)
>
>     trayIcon.show()
>     sys.exit(app.exec_())
>
> if __name__ == '__main__':
>     main()
>
> On 11/07/2016 10:49 AM, Anssi Saari wrote:
>> Demosthenes Koptsis <demosthenesk at gmail.com> writes:
>>
>>> Hello, i have a PyQT systray app with a menu and two actions.
>>>
>>> Action1 is Exit and action2 display a MessageBox with Hello World 
>>> message.
>>>
>>> When i click OK to MessageBox app quits...why?
>>>
>>> http://pastebin.com/bVA49k1C
>> I haven't done anything with Qt in a while but apparently you need to
>> call QtGui.QApplication.setQuitOnLastWindowClosed(False) before
>> trayIcon.show().
>



From dpalao.python at gmail.com  Mon Nov  7 06:42:15 2016
From: dpalao.python at gmail.com (David Palao)
Date: Mon, 7 Nov 2016 12:42:15 +0100
Subject: constructor classmethods
In-Reply-To: <0dd29885-a328-4800-88f5-51f46cfa469d@googlegroups.com>
References: <7f7156ed-e396-460e-8eb1-fb6b17bab069@googlegroups.com>
 <02dc88d9-0fd3-416b-8aac-eb2b530d0320@googlegroups.com>
 <581B4D88.9010803@stoneleaf.us>
 <mailman.18.1478184334.7975.python-list@python.org>
 <0dd29885-a328-4800-88f5-51f46cfa469d@googlegroups.com>
Message-ID: <CAKUKWzmt=EVm6w54Q6WjBMAf7tk1qNKwSQC7CtXJyQtNB8V97A@mail.gmail.com>

>>
>> If the class in question has legitimate, non-testing, reasons to specify different Queues, then make it a default argument instead:
>>
>> def __init__(self, ..., queue=None):
>>      if queue is None:
>>          queue = Queue()
>>      self.queue = queue
>
> I already stated that this is is fine, as long as the number of arguments stays in manageable levels. Although I do think testing is good enough reason to have it injected anytime. For consistency, it makes sense to have same way to create all objects. I wouldn't suggested of using that mechanism in public API's, just in internal components.
>

But if the number of arguments is not manageable, you need to change
the design anyway, for the sake of cleanness. So YAGNI.


From ethan at stoneleaf.us  Mon Nov  7 13:13:39 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 07 Nov 2016 10:13:39 -0800
Subject: constructor classmethods
In-Reply-To: <0dd29885-a328-4800-88f5-51f46cfa469d@googlegroups.com>
References: <7f7156ed-e396-460e-8eb1-fb6b17bab069@googlegroups.com>
 <02dc88d9-0fd3-416b-8aac-eb2b530d0320@googlegroups.com>
 <581B4D88.9010803@stoneleaf.us>
 <mailman.18.1478184334.7975.python-list@python.org>
 <0dd29885-a328-4800-88f5-51f46cfa469d@googlegroups.com>
Message-ID: <5820C453.1010101@stoneleaf.us>

On 11/07/2016 12:46 AM, teppo.pera at gmail.com wrote:
> torstai 3. marraskuuta 2016 14.45.49 UTC Ethan Furman kirjoitti:
>> On 11/03/2016 01:50 AM, teppo wrote:
>>
>>> The guide is written in c++ in mind, yet the concepts stands for any
>>>   programming language really. Read it through and think about it. If
>>>   you come back to this topic and say: "yeah, but it's c++", then you
>>>   haven't understood it.
>>
>> The ideas (loose coupling, easy testing) are certainly applicable in
>>  Python -- the specific methods talked about in that paper, however,
>>  are not.
>
> Please elaborate. Which ones explicitly?

The ones in place solely to make testing easier.

Others, such as passing an Engine into Car instead of making Car create its own, are valid.  To compare to what I said elsewhere, the exact Engine used is *not* an implementation detail -- any particular Car could have a range of Engines that work with it, and which is used is not determined by the car itself.

>> To go back to the original example:
>>
>> def __init__(self, ...):
>>       self.queue = Queue()
>>
>> we have several different (easy!) ways to do dependency injection:
>>
>> * inject a mock Queue into the module
>> * make queue a default parameter
>>
>> If it's just testing, go with the first option:
>>
>> import the_module_to_test
>> the_module_to_test.Queue = MockQueue
>>
>> and away you go.
>
> This is doable, but how would you inject queue (if we use Queue as an
>  example) with different variations, such as full, empty, half-full,
>  half-empty. :) For different tests.

In this case I would go with my last example:

ex = Example()
test_queue = generate_half_empty_queue()
ex.queue = test_queue
# do the testing

>> If the class in question has legitimate, non-testing, reasons to specify different Queues, then make it a default argument instead:
>>
>> def __init__(self, ..., queue=None):
>>       if queue is None:
>>           queue = Queue()
>>       self.queue = queue
>
> I already stated that this is is fine, as long as the number of arguments
>  stays in manageable levels.

How is having 15 arguments in a .create() method better than having 15 arguments in __init__() ?

>  Although I do think testing is good enough
>  reason to have it injected anytime. For consistency, it makes sense to
>  have same way to create all objects. I wouldn't suggested of using that
>  mechanism in public API's, just in internal components.

And that consistent way is to just call the class -- not to call class.create().

>> or, if it's just for testing but you don't want to hassle injecting a
>>  MockQueue into the module itself:
>>
>> def __init__(self, ..., _queue=None):
>>       if _queue is None:
>>           _queue = Queue()
>>       self.queue = _queue
>>
>> or, if the queue is only initialized (and not used) during __init__
>>  (so you can replace it after construction with no worries):
>>
>> class Example:
>>       def __init__(self, ...):
>>           self.queue = Queue()
>>
>> ex = Example()
>> ex.queue = MockQueue()
>> # proceed with test
>
> This I wouldn't recommend. It generates useless work when things start to change,
>  especially in large code bases. Don't touch internal stuff of class in tests (or
>  anywhere else).

So, if you use the create() method, and it sets up internal data structures, how do you test them?  In other words, if create() makes that queue then how do you test with a half-empty queue?

>> The thing each of those possibilities have in common is that the normal use-case
>>  of just creating the thing and moving on is the very simple:
>>
>> my_obj = Example(...)
>>
>> To sum up:  your concerns are valid, but using c++ (and many other language)
>>  idioms in Python does not make good Python code.
>
> This is not necessarily simply a c++ idiom, but a factory method design pattern [...]

Not all design patterns make sense in every language.

--
~Ethan~


From jladasky at itu.edu  Mon Nov  7 13:47:25 2016
From: jladasky at itu.edu (jladasky at itu.edu)
Date: Mon, 7 Nov 2016 10:47:25 -0800 (PST)
Subject: [Theory] How to speed up python code execution / pypy vs GPU
In-Reply-To: <581e89dd$0$1620$c3e8da3$5496439d@news.astraweb.com>
References: <nvl3q7$vut$1@news.icm.edu.pl>
 <581e25b4$0$1607$c3e8da3$5496439d@news.astraweb.com>
 <87pom9eids.fsf@bsb.me.uk> <nvllqc$1mf$1@news.icm.edu.pl>
 <581e89dd$0$1620$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <be3ca06a-6cf0-4d98-ad27-96260cdcd81c@googlegroups.com>

On Saturday, November 5, 2016 at 6:39:52 PM UTC-7, Steve D'Aprano wrote:
> On Sun, 6 Nov 2016 09:17 am, Mr. Wrobel wrote:
> 
> 
> I don't have any experience with GPU processing. I expect that it will be
> useful for somethings, but for number-crushing and numeric work, I am
> concerned that GPUs rarely provide correctly rounded IEEE-754 maths. That
> means that they are accurate enough for games where a few visual glitches
> don't matter, but they risk being inaccurate for serious work.
> 
> I fear that doing numeric work in GPUs will be returning to the 1970s, when
> every computer was incompatible with every other computer, and it was
> almost impossible to write cross-platform, correct, accurate numeric code.

Hi Steve,

You, Jason Swails, myself, and several others had a discussion about the state of GPU arithmetic and IEEE-754 compliance just over a year ago.

https://groups.google.com/forum/#!msg/comp.lang.python/Gt_FzFlES8A/r_3dbW5XzfkJ;context-place=forum/comp.lang.python

It has been very important for the field of computational molecular dynamics (and probably several other fields) to get floating-point arithmetic working right on GPU architecture.  I don't know anything about other manufacturers of GPU's, but NVidia announced IEEE-754, double-precision arithmetic for their GPU's in 2008, and it's been included in the standard since CUDA 2.0.

If floating-point math wasn't working on GPU's, I suspect that a lot of people in the scientific community would be complaining.  

Do you have any new information that would lead you to doubt what we said in the discussion we had last year?


From skip.montanaro at gmail.com  Mon Nov  7 14:27:48 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Mon, 7 Nov 2016 13:27:48 -0600
Subject: Lua tutorial help for Python programmer?
Message-ID: <CANc-5Ux3Hmptj1aDfu6+cS+kz2tfGxJ6QiqQsb5gPGwDcqSVHw@mail.gmail.com>

I just got Lua scripting dumped in my lap as a way to do some server
side scripting in Redis. The very most basic stuff isn't too hard (i =
1, a = {"x"=4, ...}, for i = 1,10,2 do ... end), but as soon as I get
beyond that, I find it difficult to formulate questions which coax
Google into useful suggestions. Is there an equivalent to the
python-tutor, python-help, or even this (python-list/comp.lang.python)
for people to ask Lua questions from the perspective of a Python
programmer? Maybe an idiom translation table?

A couple questions to show what sort of (terribly basic) stuff I'm after.

1. print(tbl) where tbl is a Lua table prints something useless like

table: 0x3c73310

How can I print a table in one go so I see all its keys and values?

2. The redis-py package helpfully converts the result of HGETALL to a
Python dictionary. On the server, The Lua code just sees an
interleaved list (array?) of the key/value pairs, e.g., "a" "1" "b"
"2" "c" "hello". I'd dictify that in Python easily enough:

dict(zip(result[::2], result[1::2]))

and get {"a": "1", "b": "2", "c": "hello"}

Skimming the Lua reference manual, I didn't see anything like dict()
and zip(). I suspect I'm thinking like a Python programmer when I
shouldn't be. Is there a Lua idiom which tackles this problem in a
straightforward manner, short of a numeric for loop?

As you can see, this is pretty trivial stuff, mostly representing
things which are just above the level of the simplest tutorial.

Thanks,

Skip


From amorawski at magna-power.com  Mon Nov  7 16:27:19 2016
From: amorawski at magna-power.com (Adam M)
Date: Mon, 7 Nov 2016 13:27:19 -0800 (PST)
Subject: [Theory] How to speed up python code execution / pypy vs GPU
In-Reply-To: <581e7ee6$0$1608$c3e8da3$5496439d@news.astraweb.com>
References: <nvl3q7$vut$1@news.icm.edu.pl>
 <581e25b4$0$1607$c3e8da3$5496439d@news.astraweb.com>
 <87pom9eids.fsf@bsb.me.uk>
 <581e7ee6$0$1608$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <4f8300f6-367e-4791-9cf2-70360088ce5d@googlegroups.com>

On Saturday, November 5, 2016 at 8:58:36 PM UTC-4, Steve D'Aprano wrote:
> On Sun, 6 Nov 2016 08:17 am, Ben Bacarisse wrote:
> 
> > Steve D'Aprano <ste... at pearwood.info> writes:
> 
> >> Here's the same program in Objective C:
> >>
> >> --- cut ---
> >>
> >> #import <Foundation/Foundation.h>
> >>
> >> int main (int argc, const char * argv[])
> >> {
> >>         NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
> >>         NSLog (@"Hello, World!");
> >>         [pool drain];
> >>         return 0;
> >> }
> >>
> >> --- cut ---
> >>
> >> Which would you rather write?
> > 
> > That's a rather odd comparison.  Why not
> > 
> >   #import <stdio.h>
> > 
> >   int main()
> >   {
> >     printf("Hello world\n");
> >     return 0;
> >   }
> 
> 
> Because that's not Objective-C? (This is not a rhetorical question.)
> 
> I'm not an Objective-C expert, but to my eye, that doesn't look like
> Objective-C. It looks like plain old regular C.
> 
> Here's where I stole the code from:
> 
> https://www.binpress.com/tutorial/objectivec-lesson-1-hello-world/41
> 
> and its not too dissimilar from the versions here:
> 
> http://rosettacode.org/wiki/Hello_world/Text#Objective-C
> 
> 
> > ?  It's decades since I wrote any Objective-C (and then not much) but I
> > think this is the closest comparison.
> 
> 
> 
> -- 
> Steve
> ?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
> enough, things got worse.


It is Objective-C. You are mistaken taking NS extensions to function names as a part of Objective-C. There are not. It is from NextStep/Sun implementation. Because they are always used a lot of people tend to think that they are part of Objective-C - but they are not - they are just libraries.


From breamoreboy at gmail.com  Mon Nov  7 16:38:19 2016
From: breamoreboy at gmail.com (breamoreboy at gmail.com)
Date: Mon, 7 Nov 2016 13:38:19 -0800 (PST)
Subject: Why are there so many Python Installers? Windows only :)
Message-ID: <a9ed297a-9ab0-488f-a79d-d61a38e9529e@googlegroups.com>

Hi folks, an interesting blog from Steve Dower giving the history of the little beasties http://stevedower.id.au/blog/why-so-many-python-installers/

Kindest regards.

Mark Lawrence.


From torriem at gmail.com  Mon Nov  7 19:26:14 2016
From: torriem at gmail.com (Michael Torrie)
Date: Mon, 7 Nov 2016 17:26:14 -0700
Subject: [Theory] How to speed up python code execution / pypy vs GPU
In-Reply-To: <nvl3q7$vut$1@news.icm.edu.pl>
References: <nvl3q7$vut$1@news.icm.edu.pl>
Message-ID: <3ad185ad-6524-aa05-64aa-58687237ee16@gmail.com>

On 11/05/2016 11:10 AM, Mr. Wrobel wrote:
> Hi,
> 
> Some skeptics asked my why there is a reason to use Python against of 
> any other "not interpreted" languages, like objective-C. As my 
> explanation, I have answered that there is a a lot of useful APIs, 
> language is modern, has advanced objective architecture, and what is the 
> most important - it is  dynamic and support is simply great.
> 
> However the same skeptics told my that, ok we believe that it is true, 
> however the code execution is much slower than any other compiled language.
> 
> I must tell you that is the reason I started to dig into internet and 
> searching some methods to speed up python's code.

I'm reminded of the old adage, premature optimization is the root of all
evil.  Trying to find ways of speeding up Python code is interesting,
but it may be less helpful to the general cases than you think.

It's undeniable that given a particular CPU-bound algorithm implemented
in Objective C and Python that the ObjC version will finish faster.
Probably an order of magnitude faster.  But even in this situation, the
question is, does it matter?  And in almost all cases, 90% of the
execution time of a program is spent in 10% of the code.  Once you
isolate that 10%, you can bring other tools to bear, even while you use
Python. For example that critical 10% could be compiled to binary code
with Cython.  Or you may find that using a library such as numpy to do
fast linear algebra is the way to go.

Many modern tasks are actually IO-bound rather than CPU bound so Python
would be just as fast as any other language. That's why Python is
well-placed in web development where scaling involves more than simply
CPU execution time.

I don't think you will have any luck in persuading your colleagues to
replace Objective-C with Python by chasing ways of speeding up Python.
But you may be able to show them the places where Python really shines
and they may come around to the idea of using Python in certain places
more often.


From steve+python at pearwood.info  Mon Nov  7 20:23:07 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Tue, 08 Nov 2016 12:23:07 +1100
Subject: [Theory] How to speed up python code execution / pypy vs GPU
References: <nvl3q7$vut$1@news.icm.edu.pl>
 <581e25b4$0$1607$c3e8da3$5496439d@news.astraweb.com>
 <87pom9eids.fsf@bsb.me.uk> <nvllqc$1mf$1@news.icm.edu.pl>
 <581e89dd$0$1620$c3e8da3$5496439d@news.astraweb.com>
 <be3ca06a-6cf0-4d98-ad27-96260cdcd81c@googlegroups.com>
Message-ID: <582128fe$0$1607$c3e8da3$5496439d@news.astraweb.com>

On Tue, 8 Nov 2016 05:47 am, jladasky at itu.edu wrote:

> On Saturday, November 5, 2016 at 6:39:52 PM UTC-7, Steve D'Aprano wrote:
>> On Sun, 6 Nov 2016 09:17 am, Mr. Wrobel wrote:
>> 
>> 
>> I don't have any experience with GPU processing. I expect that it will be
>> useful for somethings, but for number-crushing and numeric work, I am
>> concerned that GPUs rarely provide correctly rounded IEEE-754 maths. That
>> means that they are accurate enough for games where a few visual glitches
>> don't matter, but they risk being inaccurate for serious work.
>> 
>> I fear that doing numeric work in GPUs will be returning to the 1970s,
>> when every computer was incompatible with every other computer, and it
>> was almost impossible to write cross-platform, correct, accurate numeric
>> code.
> 
> Hi Steve,
> 
> You, Jason Swails, myself, and several others had a discussion about the
> state of GPU arithmetic and IEEE-754 compliance just over a year ago.

I don't know why you think I was part of this discussion -- I made one
comment early in the thread, and took part in none of the subsequent
comments.

If I had read any of the subsequent comments in the thread, I don't remember
them.

>
https://groups.google.com/forum/#!msg/comp.lang.python/Gt_FzFlES8A/r_3dbW5XzfkJ;context-place=forum/comp.lang.python


For those who dislike GoogleGroups, here's the official archive:

https://mail.python.org/pipermail/python-list/2015-February/686683.html

 
> It has been very important for the field of computational molecular
> dynamics (and probably several other fields) to get floating-point
> arithmetic working right on GPU architecture.  I don't know anything about
> other manufacturers of GPU's, but NVidia announced IEEE-754,
> double-precision arithmetic for their GPU's in 2008, and it's been
> included in the standard since CUDA 2.0.

That's excellent news, and well-done to NVidia.

But as far as I know, they're not the only manufacturer of GPUs, and they
are the only ones who support IEEE 754. So this is *exactly* the situation
I feared: incompatible GPUs with varying support for IEEE 754 making it
difficult or impossible to write correct numeric code across GPU platforms.

Perhaps it doesn't matter? Maybe people simply don't bother to use anything
but Nvidia GPUs for numeric computation, and treat the other GPUs as toys
only suitable for games.


> If floating-point math wasn't working on GPU's, I suspect that a lot of
> people in the scientific community would be complaining.

I don't.

These are scientists, not computational mathematics computer scientists. In
the 1980s, the authors of the "Numeric Recipes in ..." books, William H
Press et al, wrote a comment about the large number of scientific papers
and simulations which should be invalidated due to poor numeric properties
of the default pseudo-random number generators available at the time.

I see no reason to think that the numeric programming sophistication of the
average working scientist or Ph.D. student has improved since then.

The average scientist cannot even be trusted to write an Excel spreadsheet
without errors that invalidate their conclusion:

https://www.washingtonpost.com/news/wonk/wp/2016/08/26/an-alarming-number-of-scientific-papers-contain-excel-errors/

let alone complex floating point numeric code. Sometimes those errors can
change history: the best, some might say *only*, evidence for the austerity
policies which have been destroying the economies in Europe for almost a
decade now is simply a programming error.

http://www.bloomberg.com/news/articles/2013-04-18/faq-reinhart-rogoff-and-the-excel-error-that-changed-history

These are not new problems: dubious numeric computations have plagued
scientists and engineers for decades, there is still a huge publication
bias against negative results, most papers are written but not read, and
even those which are read, most are wrong.

http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124

Especially in fast moving fields of science where there is money to be made,
like medicine and genetics. There the problems are much, much worse.

Bottom line: I'm very glad that Nvidia now support IEEE 754 maths, and that
reduces my concerns: at least users of one common GPU can be expected to
have correctly rounded results of basic arithmetic operations.


-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From and.damore at gmail.com  Tue Nov  8 03:12:30 2016
From: and.damore at gmail.com (Andrea D'Amore)
Date: Tue, 8 Nov 2016 09:12:30 +0100
Subject: Lua tutorial help for Python programmer?
In-Reply-To: <CANc-5Ux3Hmptj1aDfu6+cS+kz2tfGxJ6QiqQsb5gPGwDcqSVHw@mail.gmail.com>
References: <CANc-5Ux3Hmptj1aDfu6+cS+kz2tfGxJ6QiqQsb5gPGwDcqSVHw@mail.gmail.com>
Message-ID: <CAMonFrSv0hrxZ+05FtBcSb9TEOZJPfqQ_eL_u2y-PL7tfUhjiA@mail.gmail.com>

On 7 November 2016 at 20:27, Skip Montanaro <skip.montanaro at gmail.com> wrote:
> I just got Lua scripting dumped in my lap as a way to do some server
> side scripting in Redis. The very most basic stuff isn't too hard (i =
> 1, a = {"x"=4, ...}, for i = 1,10,2 do ... end), but as soon as I get
> beyond that, I find it difficult to formulate questions which coax
> Google into useful suggestions. Is there an equivalent to the
> python-tutor, python-help, or even this (python-list/comp.lang.python)
> for people to ask Lua questions from the perspective of a Python
> programmer? Maybe an idiom translation table?

There's lua-list, I figure all your questions fit better there than here.
There's the official wiki on lua-users [1], I don't know about a
Py-Lua Rosetta Stone.


> 1. print(tbl) where tbl is a Lua table prints something useless like
[?]
> How can I print a table in one go so I see all its keys and values?

Use the pairs() iterator function (check the reference manual for
ipairs() as well):

    for key, value in pairs(my_table) do
        print(key, value)
    end


> 2. The redis-py package helpfully converts the result of HGETALL to a
> Python dictionary. On the server, The Lua code just sees an
> interleaved list (array?) of the key/value pairs, e.g., "a" "1" "b"
> "2" "c" "hello". I'd dictify that in Python easily enough:
[?]
> Skimming the Lua reference manual, I didn't see anything like dict()
> and zip().

IIRC tables are the only data structures in Lua, actually I liked this
simplification very much.


> I suspect I'm thinking like a Python programmer when I
> shouldn't be. Is there a Lua idiom which tackles this problem in a
> straightforward manner, short of a numeric for loop?


IIRC the standard library is quite compact, so no.
If you want something like more straightforward than

    dict = {}
    for i = 1, #results, 2 do
        dict[results[i]] = results[i+1]
    end

You can define your own iterator function and have "for key, value in
?" in the for loop. Not sure it's worth it.

Beware that I'm no Lua expert, I just liked the language and read
about it but never actually used in any project. I suggesting checking
the mailing list or the IRC channel.


> As you can see, this is pretty trivial stuff, mostly representing
> things which are just above the level of the simplest tutorial.

Check "Programming in Lua" book, older versions are made available
online by the author.


[1]: http://lua-users.org/wiki/TutorialDirectory

-- 
Andrea


From info at egenix.com  Tue Nov  8 05:22:25 2016
From: info at egenix.com (eGenix Team: M.-A. Lemburg)
Date: Tue, 8 Nov 2016 11:22:25 +0100
Subject: ANN: PyDDF Python Sprint 2016
Message-ID: <5821A761.6000300@egenix.com>

[This announcement is in German since it targets a Python sprint in
 D?sseldorf, Germany]

________________________________________________________________________
ANK?NDIGUNG

		       PyDDF Python Sprint 2016
			    in D?sseldorf

		 Samstag, 19.11.2016, 10:00-18:00 Uhr
		 Sonntag, 20.11.2016, 10:00-18:00 Uhr
	trivago GmbH,  Karl-Arnold-Platz 1A,  40474 D?sseldorf


		      Python Meeting D?sseldorf
		     http://pyddf.de/sprint2016/

________________________________________________________________________
INFORMATION

Das Python Meeting D?sseldorf (PyDDF) veranstaltet mit freundlicher
Unterst?tzung der *trivago GmbH* ein Python Sprint Wochenende im
September.

Der Sprint findet am Wochenende 19./20.11.2016 in der trivago
Niederlassung am Karl-Arnold-Platz 1A statt (nicht am Bennigsen-Platz
1). Bitte beim Pf?rtner melden.

Google Maps:
https://www.google.de/maps/dir/51.2452741,6.7711581//@51.2450432,6.7714612,18.17z?hl=de

Folgende Themengebiete haben wir als Anregung angedacht:

 * Openpyxl

   Openpyxl ist eine Python Bibliothek, mit der man Excel 2010+
   Dateien lesen und schreiben kann.

   Charlie ist Co-Maintainer des Pakets.

 * MicroPython auf ESP8266 und BBC micro:bit

   MicroPython ist eine Python 3 Implementierung f?r Micro
   Controller. Sie l?uft u.a. auf dem BBC micro:bit, einem
   Ein-Patinen-Computer, der in Gro?britanien an Kinder der 7. Klassen
   verteilt wurde, und dem mittlerweile sehr popul?ren IoT Chip
   ESP8266, der WLAN unterst?tzt.

   Im Sprint wollen wir versuchen, ein Mesh Network aus BBC micro:bits
   aufzubauen, das dann an einen ESP8266 mit dem WLAN verbunden
   wird. Alles mit Hilfe von MicroPython.

   Vorkenntnisse sind eigentlich keine n?tig. Wir werden mindestens
   einen ESP8266 und drei BBC micro:bits zur Verf?gung haben.

Nat?rlich kann jeder Teilnehmer weitere Themen vorschlagen, z.B.

 * Kivy
 * Raspberry Pi
 * FritzConnection
 * OpenCV
 * u.a.

Alles weitere und die Anmeldung findet Ihr auf der Sprint Seite:

    http://pyddf.de/sprint2016/

Teilnehmer sollten sich zudem auf der PyDDF Liste anmelden, da wir
uns dort koordinieren:

    https://www.egenix.com/mailman/listinfo/pyddf

________________________________________________________________________
?BER UNS

Das Python Meeting D?sseldorf (PyDDF) ist eine regelm??ige Veranstaltung
in D?sseldorf, die sich an Python Begeisterte aus der Region wendet:

 * http://pyddf.de/

Einen guten ?berblick ?ber die Vortr?ge bietet unser YouTube-Kanal,
auf dem wir die Vortr?ge nach den Meetings ver?ffentlichen:

 * http://www.youtube.com/pyddf/

Veranstaltet wird das Meeting von der eGenix.com GmbH, Langenfeld,
in Zusammenarbeit mit Clark Consulting & Research, D?sseldorf:

 * http://www.egenix.com/
 * http://www.clark-consulting.eu/

Mit freundlichen Gr??en,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 08 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/



From skip.montanaro at gmail.com  Tue Nov  8 13:07:00 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Tue, 8 Nov 2016 12:07:00 -0600
Subject: Lua tutorial help for Python programmer?
In-Reply-To: <CAMonFrSv0hrxZ+05FtBcSb9TEOZJPfqQ_eL_u2y-PL7tfUhjiA@mail.gmail.com>
References: <CANc-5Ux3Hmptj1aDfu6+cS+kz2tfGxJ6QiqQsb5gPGwDcqSVHw@mail.gmail.com>
 <CAMonFrSv0hrxZ+05FtBcSb9TEOZJPfqQ_eL_u2y-PL7tfUhjiA@mail.gmail.com>
Message-ID: <CANc-5UyQhd+eE04UTKvctP6sRyPKW4QK5FUgBwVAWPhTF3oitQ@mail.gmail.com>

On Tue, Nov 8, 2016 at 2:12 AM, Andrea D'Amore <and.damore at gmail.com> wrote:
> There's lua-list, I figure all your questions fit better there than here.
> There's the official wiki on lua-users [1], I don't know about a
> Py-Lua Rosetta Stone.

Thanks. I asked here specifically because I was interested in Lua from
a Python perspective. I know that I see people bring incorrect
(suboptimal?) idioms from other languages when starting with Python. I
was hoping that some Python people who have a foot in the Lua world
could help me avoid that mistake.

Skip


From teppo.pera at gmail.com  Tue Nov  8 18:01:32 2016
From: teppo.pera at gmail.com (teppo.pera at gmail.com)
Date: Tue, 8 Nov 2016 15:01:32 -0800 (PST)
Subject: constructor classmethods
In-Reply-To: <mailman.35.1478542412.30068.python-list@python.org>
References: <7f7156ed-e396-460e-8eb1-fb6b17bab069@googlegroups.com>
 <02dc88d9-0fd3-416b-8aac-eb2b530d0320@googlegroups.com>
 <581B4D88.9010803@stoneleaf.us>
 <mailman.18.1478184334.7975.python-list@python.org>
 <0dd29885-a328-4800-88f5-51f46cfa469d@googlegroups.com>
 <5820C453.1010101@stoneleaf.us>
 <mailman.35.1478542412.30068.python-list@python.org>
Message-ID: <9bc2c08c-b5d2-4ae2-a2e6-8d939b5dccee@googlegroups.com>

> How is having 15 arguments in a .create() method better than having 15 arguments in __init__() ?
> So, if you use the create() method, and it sets up internal data structures, how do you test them?  In other words, if create() makes that queue then how do you test with a half-empty queue?
> Not all design patterns make sense in every language.
 
Seems that there is still some unclarity about the whole proposal, so I combine your questions into an example. But first, little more background.

Generally, with testing, it would be optimal to test outputs of the system for given inputs without caring how things are implemented. That way, any changes in implementation won't affect test results. Very trivial example would be something like this:

def do_something_important(input1, input2, input3)
    return  # something done with input1, input2, input3

Implementation of do_something_important can be one liner, or it can contain multiple classes, yet the result of the function is what matters. That's also the basic strategy (one of many) I try to follow with when testing the code I write.

Now, testing the class using queue (as an example) should follow same pattern, thus a simple example how to test it would look like this (assuming that everyone knows how to use mock library):

# This is just an example testing with DI and mocks.

class Example:
   def __init__(self, queue):
       self._queue = queue

   def can_add(self):
       return not self._queue.full()  

def TestExample(unittest.TestCase):
    def setUp(self):
        self.queue = mock.MagicMock()

    def test_it_should_be_possible_to_know_when_there_is_still_room_for_items(self):
       self.queue.full.return_value = False
       example = Example(self.queue)
       self.assertTrue(example.can_add())

    def test_it_should_be_possible_to_know_when_no_more_items_can_be_added(self):
       self.queue.full.return_value = True
       example = Example(self.queue)
       self.assertFalse(example.can_add())

In above example, example doesn't really care what class the object is. Only full method is needed to be implemented. Injected class can be Queue, VeryFastQueue or LazyQueue, as long as they implement method "full" (duck-typing!). Test takes advantage of that and changing the implementation won't break the tests (tests are not caring how Example is storing the Queue). 
 Also, adding more cases is trivial and should also make think the actual implementation and what is needed to be taken care of. For example, self.queue.full.return_value = None, or self.queue.full.side_effect = ValueError(). How should code react on those?

Then comes the next step, doing the actual DI. One solution is:

class Example:
    def __init__(self, queue=None):
        self._queue = queue or Queue()

Fine approach, but technically __init__ has two execution branches and someone staring blindly coverages might require covering those too. Then we can use class method too.

class Example:
    def __init__(self, queue):
        self._queue = queue

    @classmethod
    def create(cls):
        q = Queue()
        # populate_with_defaults
        # Maybe get something from db too for queue...
        return cls(q)

As said, create-method is for convenience. it can (and should) contain minimum set of arguments needed from user (no need to be 15 even if __init__ would require it) to create the object. It creates the fully functioning Example object with default dependencies. Do notice that tests I wrote earlier would still work. Create can contain slow executing code, if needed, but it won't slow down testing the Example class itself. 

Finally, if you want to be tricky and write own decorator for object construction, Python would allow you to do that.

@spec('queue')  # generates __init__ that populates instance with queue given as arg
class Example:
    @classmethod
    def create(cls):
        return cls(Queue())

Example can still be initialized calling Example(some_dependency), or calling Example.create() which provides default configuration. Writing the decorator would give unlimited ways to extend the class. And test written in the beginning of the post would still pass.


From rosuav at gmail.com  Tue Nov  8 20:15:46 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 9 Nov 2016 12:15:46 +1100
Subject: constructor classmethods
In-Reply-To: <9bc2c08c-b5d2-4ae2-a2e6-8d939b5dccee@googlegroups.com>
References: <7f7156ed-e396-460e-8eb1-fb6b17bab069@googlegroups.com>
 <02dc88d9-0fd3-416b-8aac-eb2b530d0320@googlegroups.com>
 <581B4D88.9010803@stoneleaf.us>
 <mailman.18.1478184334.7975.python-list@python.org>
 <0dd29885-a328-4800-88f5-51f46cfa469d@googlegroups.com>
 <5820C453.1010101@stoneleaf.us>
 <mailman.35.1478542412.30068.python-list@python.org>
 <9bc2c08c-b5d2-4ae2-a2e6-8d939b5dccee@googlegroups.com>
Message-ID: <CAPTjJmrZmeNS7HqJ2RuzEumdboauS8o_Ukqpb-17s14m0wp_Ag@mail.gmail.com>

On Wed, Nov 9, 2016 at 10:01 AM,  <teppo.pera at gmail.com> wrote:
> One solution is:
>
> class Example:
>     def __init__(self, queue=None):
>         self._queue = queue or Queue()
>
> Fine approach, but technically __init__ has two execution branches and someone staring blindly coverages might require covering those too. Then we can use class method too.
>
> class Example:
>     def __init__(self, queue):
>         self._queue = queue
>
>     @classmethod
>     def create(cls):
>         q = Queue()
>         # populate_with_defaults
>         # Maybe get something from db too for queue...
>         return cls(q)
>
> As said, create-method is for convenience. it can (and should) contain minimum set of arguments needed from user (no need to be 15 even if __init__ would require it) to create the object.
>

You gain nothing, though. Whether your code paths are in create() or
in __init__, you still have them. You can make __init__ take no
mandatory arguments (other than self) and then it's still just as easy
to use. Tell me, without looking it up: How many arguments does the
built-in open() function take? But you don't have to worry about them,
most of the time.

Python has idioms available that C++ simply can't use, so what's right
for C++ might well not be right for Python, simply because there's
something better.

ChrisA


From steve+python at pearwood.info  Tue Nov  8 21:25:37 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Wed, 09 Nov 2016 13:25:37 +1100
Subject: constructor classmethods
References: <7f7156ed-e396-460e-8eb1-fb6b17bab069@googlegroups.com>
 <02dc88d9-0fd3-416b-8aac-eb2b530d0320@googlegroups.com>
 <581B4D88.9010803@stoneleaf.us>
 <mailman.18.1478184334.7975.python-list@python.org>
 <0dd29885-a328-4800-88f5-51f46cfa469d@googlegroups.com>
 <5820C453.1010101@stoneleaf.us>
 <mailman.35.1478542412.30068.python-list@python.org>
 <9bc2c08c-b5d2-4ae2-a2e6-8d939b5dccee@googlegroups.com>
Message-ID: <58228922$0$1599$c3e8da3$5496439d@news.astraweb.com>

On Wed, 9 Nov 2016 10:01 am, teppo.pera at gmail.com wrote:

> Generally, with testing, it would be optimal to test outputs of the system
> for given inputs without caring how things are implemented.

I disagree with that statement.

You are talking about "black-box testing" -- the test code should treat the
code being tested as a completely opaque, black box where the inner
workings are invisible. Tests are only written against the interface.

The alternative is "white-box testing", where the test code knows exactly
what the implementation is, and can test against the implementation, not
just the interface.

White-box testing is more costly, because any change in implementation will
cause a lot of code churn in the tests. Change the implementation, and
tests will disappear. Change it back, and the tests need to be reverted.

But that churn only applies to one person: the maintainer of the test. It
doesn't affect users of the code. It is certainly a cost, but it is
narrowly focused on the maintainer, not the users.

White-box testing gives superior test coverage, because the test author is
aware of the implementation. Let's suppose that I was writing some
black-box tests for Python's list.sort() method. Knowing nothing of the
implementation, I might think that there's only a handful of cases I need
to care about:

- an empty list []
- a single item list [1]
- an already sorted list [1, 2, 3, 4, 5]
- a list sorted in reverse order [5, 4, 3, 2, 1]
- a few examples of unsorted lists, e.g. [3, 5, 2, 4, 1]

And we're done! Black-box testing makes tests easy.

But in fact, that test suite is completely insufficient. The implementation
of list.sort() uses two different sort algorithms: insertion sort for short
lists, and Timsort for long lists. My black-box test suite utterly fails to
test Timsort.

To be sufficient, I *must* test both insertion sort and Timsort, and I can
only guarantee to do that by testing against the implementation, not the
interface.

Black-box testing is better than nothing, but white-box testing is much more
effective.


> That way, any 
> changes in implementation won't affect test results. Very trivial example
> would be something like this:
> 
> def do_something_important(input1, input2, input3)
>     return  # something done with input1, input2, input3
> 
> Implementation of do_something_important can be one liner, or it can
> contain multiple classes, yet the result of the function is what matters.

Certainly. And if you test do_something_important against EVERY possible
combination of inputs, then you don't need to care about the
implementation, since you've tested every single possible case.

But how often do you do that?


[...]
> Then comes the next step, doing the actual DI. One solution is:
> 
> class Example:
>     def __init__(self, queue=None):
>         self._queue = queue or Queue()

That's buggy. If I pass a queue which is falsey, you replace it with your
own default queue instead of the one I gave you. That's wrong.

If I use queue.Queue, that's not a problem, because empty queues are still
truthy. But if I use a different queue implementation, then your code
breaks.

The lessen here is: when you want to test for None, TEST FOR NONE. Don't
use "or" when you mean "if obj is None".


> Fine approach, but technically __init__ has two execution branches and
> someone staring blindly coverages might require covering those too. 

And here we see the advantage of white-box testing. We do need tests for
both cases: to ensure that the given queue is always used, and that a new
Queue is only used when no queue was given at all. A pure blackbox tester
might not imagine the need for these two cases, as he is not thinking about
the implementation.



> Then we can use class method too.
> 
> class Example:
>     def __init__(self, queue):
>         self._queue = queue
> 
>     @classmethod
>     def create(cls):
>         q = Queue()
>         # populate_with_defaults
>         # Maybe get something from db too for queue...
>         return cls(q)
> 
> As said, create-method is for convenience. it can (and should) contain
> minimum set of arguments needed from user (no need to be 15 even if
> __init__ would require it) to create the object. 

Why would __init__ require fifteen arguments if the user can pass one
argument and have the other fourteen filled in by default?

The question here is, *why* is the create() method a required part of your
API? There's no advantage to such a change of spelling. The Python style is
to spell instance creation:

    instance = MyClass(args)

not 

    instance = MyClass.create(args)

Of course you can design your classes using any API you like:

    instance = MyClass.Make_Builder().build().create_factory().create(args)

if you insist. But if all create() does is fill in some default values for
you, then it is redundant. The __init__ method can just as easily fill in
the default values. All you are doing is changing the spelling:

    MyClass(arg)  # insert ".create" before the left parenthesis

and that's just being different for the sake of being different:

    list.sort()  # insert ".sorting_factory" before the left parenthesis

You're not actually adding any new functionality or new abilities, or making
testing easier. You're just increasing the amount of typing needed.


> It creates the fully 
> functioning Example object with default dependencies. Do notice that tests
> I wrote earlier would still work. Create can contain slow executing code,
> if needed, but it won't slow down testing the Example class itself.

Of course it will. You are testing the create() method aren't you? If you're
not testing it, how do you know it is working? If you are testing it, then
it will be just as slow in the tests as it will be slow for the poor users
who have to call it.

I think "slow code" is a red herring. If the code is slow, its slow whether
you put it in the __init__ or create method.

For that matter, testing is a red herring too. It doesn't matter whether
your public API is:

    Example(queue)
    Example()

or:

    Example(queue)
    Example.create()

you still have to test both cases, regardless of whether you are doing
white-box or black-box testing. These are two separate APIs:

    create an instance from a specified queue
    create an instance without a specified queue

so regardless of whether you look inside the implementation or not, you
still have to test both cases.


> Finally, if you want to be tricky and write own decorator for object
> construction, Python would allow you to do that.
> 
> @spec('queue')  # generates __init__ that populates instance with queue
> given as arg 
> class Example: 
>     @classmethod
>     def create(cls):
>         return cls(Queue())

I don't think that this decorator example makes any sense. At least I cannot
understand it. Why on earth would you write a decorator to inject an
__init__ method into a given class? Unless you have many classes with
identical __init__ methods, what's the point?




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From tim at akwebsoft.com  Tue Nov  8 22:59:19 2016
From: tim at akwebsoft.com (Tim Johnson)
Date: Tue, 8 Nov 2016 18:59:19 -0900
Subject: What is currently the recommended way to work with a
 distutils-based setup.py that requires compilation?
In-Reply-To: <9da79944-6bf3-0a65-a4ab-197f5799af4d@mail.mipt.ru>
References: <9da79944-6bf3-0a65-a4ab-197f5799af4d@mail.mipt.ru>
Message-ID: <20161109035919.GD4137@mail.akwebsoft.com>

* Ivan Pozdeev via Python-list <python-list at python.org> [161106 17:28]:
> https://wiki.python.org/moin/WindowsCompilers has now completely replaced
> instructions for `distutils`-based packages (starting with `from
> distutils.core import setup`) with ones for `setuptools`-based ones
> (starting with `from setuptools import setup`).
> 
> However, if I have a `distutils`-based `setup.py`, when I run it,
> `setuptools` is not used - thus the instructions on the page don't work.
> 
> It is possible to run a `distutils`-based script through `setuptools`, as
> `pip` does, but it requires the following code
> (https://github.com/pypa/pip/blob/8.1.2/pip/req/req_install.py#L849 ):
> 
>     python -u -c "import setuptools, tokenize;__file__=<setup.py full path>;
>         exec(compile(getattr(tokenize, 'open', open)(__file__).read()
>         .replace('\\r\\n', '\\n'), __file__, 'exec'))" <setup.py arguments>
> 
> They can't possibly expect me to type that on the command line each time,
> now can they?
Ivan, it looks like you aren't getting any answers from seasoned
list gurus to your question.

So, I'm going to take a stab at this and I hope you are not mislead
or misdirected by my comments.

> They can't possibly expect me to type that on the command line each time,

The code that you are quoting above can be placed in a script file
and executed at will. Once you get the syntax correct, you will then
be able to execute that script at any time.

I don't know what operating system you are using: Linux and Mac work
pretty much similarly when it comes to console scripts, windows will
have a different approach, but not radically so.

I hope this helps or puts you on a constructive path.

> I also asked this at http://stackoverflow.com/q/40174932/648265 a couple of
> days ago (to no avail).
> 
> -- 
> 
> Regards,
> Ivan
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
Tim 
http://www.akwebsoft.com, http://www.tj49.com


From marko at pacujo.net  Wed Nov  9 01:50:32 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Wed, 09 Nov 2016 08:50:32 +0200
Subject: constructor classmethods
References: <7f7156ed-e396-460e-8eb1-fb6b17bab069@googlegroups.com>
 <02dc88d9-0fd3-416b-8aac-eb2b530d0320@googlegroups.com>
 <581B4D88.9010803@stoneleaf.us>
 <mailman.18.1478184334.7975.python-list@python.org>
 <0dd29885-a328-4800-88f5-51f46cfa469d@googlegroups.com>
 <5820C453.1010101@stoneleaf.us>
 <mailman.35.1478542412.30068.python-list@python.org>
 <9bc2c08c-b5d2-4ae2-a2e6-8d939b5dccee@googlegroups.com>
 <58228922$0$1599$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <878tstno3r.fsf@elektro.pacujo.net>

Steve D'Aprano <steve+python at pearwood.info>:

> On Wed, 9 Nov 2016 10:01 am, teppo.pera at gmail.com wrote:
>> Generally, with testing, it would be optimal to test outputs of the
>> system for given inputs without caring how things are implemented.
>
> I disagree with that statement.

I, OTOH, agree with it.

> But in fact, that test suite is completely insufficient. The
> implementation of list.sort() uses two different sort algorithms:
> insertion sort for short lists, and Timsort for long lists. My
> black-box test suite utterly fails to test Timsort.

Independent algorithms can be packaged as independent software
components and subjected to separate black-box tests -- through their
advertised APIs.

> Black-box testing is better than nothing, but white-box testing is
> much more effective.

I've had to deal with a lot of white-box test code. In practice,

 * It relies on internal reporting by the implementation instead of
   real, observed behavior.

 * It hasn't stayed current with the implementation.

 * It has convoluted the code with scary conditional compilation and
   other tricks, thus lowering the quality of the implementation.

> Certainly. And if you test do_something_important against EVERY
> possible combination of inputs, then you don't need to care about the
> implementation, since you've tested every single possible case.

The space of every imaginable situation is virtually infinite; testing
will only scratch the surface no matter how much effort you put into it.

I'm thinking of recent bugs that have sneaked into a product. The
question was why our tests didn't catch the bugs. The answer was that
those particular sequences weren't included in the thousands of test
cases that we run regularly.

>> class Example:
>>     def __init__(self, queue=None):
>>         self._queue = queue or Queue()
>
> That's buggy. If I pass a queue which is falsey, you replace it with
> your own default queue instead of the one I gave you. That's wrong.

And your two thousand white-box test cases might miss the bug as well.

> The lessen here is: when you want to test for None, TEST FOR NONE.
> Don't use "or" when you mean "if obj is None".

In the real world, that's among the most benign of bugs that riddle
software (and software tests). Programming is too hard for mortals.

Junior software developers are full of ambitious ideas -- sky's the
limit. Experienced software developers are full of awe if anything
actually works.


Marko


From john_ladasky at sbcglobal.net  Wed Nov  9 02:35:04 2016
From: john_ladasky at sbcglobal.net (John Ladasky)
Date: Tue, 8 Nov 2016 23:35:04 -0800 (PST)
Subject: [Theory] How to speed up python code execution / pypy vs GPU
In-Reply-To: <582128fe$0$1607$c3e8da3$5496439d@news.astraweb.com>
References: <nvl3q7$vut$1@news.icm.edu.pl>
 <581e25b4$0$1607$c3e8da3$5496439d@news.astraweb.com>
 <87pom9eids.fsf@bsb.me.uk> <nvllqc$1mf$1@news.icm.edu.pl>
 <581e89dd$0$1620$c3e8da3$5496439d@news.astraweb.com>
 <be3ca06a-6cf0-4d98-ad27-96260cdcd81c@googlegroups.com>
 <582128fe$0$1607$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <f8f953ef-86c7-44c7-a12f-c734b9c32dc0@googlegroups.com>

On Monday, November 7, 2016 at 5:23:25 PM UTC-8, Steve D'Aprano wrote:
> On Tue, 8 Nov 2016 05:47 am, j... at i...edu wrote:
> > It has been very important for the field of computational molecular
> > dynamics (and probably several other fields) to get floating-point
> > arithmetic working right on GPU architecture.  I don't know anything about
> > other manufacturers of GPU's, but NVidia announced IEEE-754,
> > double-precision arithmetic for their GPU's in 2008, and it's been
> > included in the standard since CUDA 2.0.
> 
> That's excellent news, and well-done to NVidia.
> 
> But as far as I know, they're not the only manufacturer of GPUs, and they
> are the only ones who support IEEE 754. So this is *exactly* the situation
> I feared: incompatible GPUs with varying support for IEEE 754 making it
> difficult or impossible to write correct numeric code across GPU platforms.
> 
> Perhaps it doesn't matter? Maybe people simply don't bother to use anything
> but Nvidia GPUs for numeric computation, and treat the other GPUs as toys
> only suitable for games.

Maybe so.  I only know for certain that recent NVidia devices comply with IEEE-754.  Others might work too.

> > If floating-point math wasn't working on GPU's, I suspect that a lot of
> > people in the scientific community would be complaining.
> 
> I don't.
> 
> These are scientists, not computational mathematics computer scientists. In
> the 1980s, the authors of the "Numeric Recipes in ..." books, William H
> Press et al, wrote a comment about the large number of scientific papers
> and simulations which should be invalidated due to poor numeric properties
> of the default pseudo-random number generators available at the time.
> 
> I see no reason to think that the numeric programming sophistication of the
> average working scientist or Ph.D. student has improved since then.

I work a lot with a package called GROMACS, which does highly iterative calculations to simulate the motions of atoms in complex molecules.  GROMACS can be built to run on a pure-CPU platform (taking advantage of multiple cores, if you want), a pure-GPU platform (leaving your CPU cores free), or a blended platform, where certain parts of the algorithm run on CPUs and other parts on GPUs.  This latter configuration is the most powerful, because only some parts of the simulation algorithm are optimal for GPUs.  GROMACS only supports NVidia hardware with CUDA 2.0+.

Because of the iterative nature of these calculations, small discrepancies in the arithmetic algorithms can rapidly lead to a completely different-looking result.  In order to verify the integrity of GROMACS, the developers run simulations with all three supported hardware configurations, and verify that the results are identical.  Now, I don't know that every last function and corner case in the IEEE-754 suite gets exercised by GROMACS, but that's a strong vote of confidence.

> The average scientist cannot even be trusted to write an Excel spreadsheet
> without errors that invalidate their conclusion:
> 
> https://www.washingtonpost.com/news/wonk/wp/2016/08/26/an-alarming-number-of-scientific-papers-contain-excel-errors/
>
>
> let alone complex floating point numeric code. Sometimes those errors can
> change history: the best, some might say *only*, evidence for the austerity
> policies which have been destroying the economies in Europe for almost a
> decade now is simply a programming error.
> 
> http://www.bloomberg.com/news/articles/2013-04-18/faq-reinhart-rogoff-and-the-excel-error-that-changed-history

I know this story.  It's embarrassing.

> These are not new problems: dubious numeric computations have plagued
> scientists and engineers for decades, there is still a huge publication
> bias against negative results, most papers are written but not read, and
> even those which are read, most are wrong.
> 
> http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124
> 
> Especially in fast moving fields of science where there is money to be made,
> like medicine and genetics. There the problems are much, much worse.
> 
> Bottom line: I'm very glad that Nvidia now support IEEE 754 maths, and that
> reduces my concerns: at least users of one common GPU can be expected to
> have correctly rounded results of basic arithmetic operations.
> 
> 
> -- 
> Steve
> ?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
> enough, things got worse.

You're right Steve, the election results are rolling in.




From teppo.pera at gmail.com  Wed Nov  9 03:52:06 2016
From: teppo.pera at gmail.com (teppo.pera at gmail.com)
Date: Wed, 9 Nov 2016 00:52:06 -0800 (PST)
Subject: constructor classmethods
In-Reply-To: <58228922$0$1599$c3e8da3$5496439d@news.astraweb.com>
References: <7f7156ed-e396-460e-8eb1-fb6b17bab069@googlegroups.com>
 <02dc88d9-0fd3-416b-8aac-eb2b530d0320@googlegroups.com>
 <581B4D88.9010803@stoneleaf.us>
 <mailman.18.1478184334.7975.python-list@python.org>
 <0dd29885-a328-4800-88f5-51f46cfa469d@googlegroups.com>
 <5820C453.1010101@stoneleaf.us>
 <mailman.35.1478542412.30068.python-list@python.org>
 <9bc2c08c-b5d2-4ae2-a2e6-8d939b5dccee@googlegroups.com>
 <58228922$0$1599$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <a3007c85-8a61-439e-be45-0e0c1db73d21@googlegroups.com>

keskiviikko 9. marraskuuta 2016 2.25.59 UTC Steve D'Aprano kirjoitti:
> On Wed, 9 Nov 2016 10:01 am, teppo... at gmail.com wrote:
> 
> > Generally, with testing, it would be optimal to test outputs of the system
> > for given inputs without caring how things are implemented.
> 
> I disagree with that statement.

I'm aware of differences of white-box and black-box testing. White-box is based on knowledge of code and black-box is based on knowledge of requirements. Knowledge of code doesn't mean that test code queries the internals of the code being tested. More specifically, example._queue should not ever be in test code (this is what I mean not caring about the implementation). With white-box testing, you can locate specific branch, say exception, and test that, if it affects the output of the function somehow. If not, what it is doing there?

> But that churn only applies to one person: the maintainer of the test. It 
> doesn't affect users of the code. It is certainly a cost, but it is 
> narrowly focused on the maintainer, not the users. 

What happens when the the maintainer gets sick or leaves the company? Maintainer of the tests should be team of developers. Focusing maintenance of tests to single person will lead to project disaster sooner or later and this correlates well with size of the codebase. 


> > That way, any 
> > changes in implementation won't affect test results. Very trivial example
> > would be something like this:
> > 
> > def do_something_important(input1, input2, input3)
> >     return  # something done with input1, input2, input3
> > 
> > Implementation of do_something_important can be one liner, or it can
> > contain multiple classes, yet the result of the function is what matters.
> 
> Certainly. And if you test do_something_important against EVERY possible
> combination of inputs, then you don't need to care about the
> implementation, since you've tested every single possible case.
> 
> But how often do you do that?

Not necessarily every possible combination. But probably I'd be adding new test if bug happens to sneak into production.

 
> The lessen here is: when you want to test for None, TEST FOR NONE. Don't
> use "or" when you mean "if obj is None".
Explicit test against None is good thing, but this is off-topic.

 
> Why would __init__ require fifteen arguments if the user can pass one
> argument and have the other fourteen filled in by default?
How would you fill a queue in default argument using database connection?


> The question here is, *why* is the create() method a required part of your
> API? There's no advantage to such a change of spelling. The Python style is
> to spell instance creation:
> 
>     instance = MyClass(args)
> 
> not 
> 
>     instance = MyClass.create(args)

In above case, probably no need.

If we are doing manual DI, like this:
instance = MyClass(args, deps)

then it might make sense to have
MyClass.create(args)

Code using the class won't need to deliver the dependencies manually. More on this later.


> You're not actually adding any new functionality or new abilities, or making
> testing easier. You're just increasing the amount of typing needed.

This is what I'm disagreeing with and would like to see answer or alternative equivalent suggestion.

Example class is easy to test, because I can inject there mock object in __init__ and only thing init does is assigns attributes to it's instance. It is massively better alternative than assigning to example._queue, as that would be subject to failing tests for no good reason. And things get worse when doing example._deep._deeper._deepest, when change of implementation in any of those nested dependencies will cause your tests to fail. Tests should fail when the behavior of your system changes (or is not what you expect).

How do you test above class without DI and without using privates in tests (break Law of Demeter).


> Of course it will. You are testing the create() method aren't you? If you're
> not testing it, how do you know it is working? If you are testing it, then
> it will be just as slow in the tests as it will be slow for the poor users
> who have to call it.

Sorry, I should say unit test here. I might ignore create in unit tests or do tests just to test the create, trying to mock dependencies, if it is possible without getting heart attack. Otherwise, I'd leave testing for acceptance tests, who would be using real stuff anyways and would be slow anyways. Idea is having tons of fast unit tests vs dozens of slower acceptance tests.

> you still have to test both cases, regardless of whether you are doing
> white-box or black-box testing. These are two separate APIs:
Agree, I'd just do it in different scope of testing.

> > @spec('queue')  # generates __init__ that populates instance with queue
> > given as arg 
> > class Example: 
> >     @classmethod
> >     def create(cls):
> >         return cls(Queue())
> 
> I don't think that this decorator example makes any sense. At least I cannot
> understand it. Why on earth would you write a decorator to inject an
> __init__ method into a given class? Unless you have many classes with
> identical __init__ methods, what's the point?

@spec('queue') would generate __init__(self, queue): self._queue = queue
@spec('queue', 'list') would generate 
__init__(self, queue, list): 
    self._queue = queue
    self._list = list
etc...

Similar mechanism is used in pinject, which is one of the DI frameworks to Python.

Br,
Teppo


From auriocus at gmx.de  Wed Nov  9 03:59:36 2016
From: auriocus at gmx.de (Christian Gollwitzer)
Date: Wed, 9 Nov 2016 09:59:36 +0100
Subject: [Theory] How to speed up python code execution / pypy vs GPU
In-Reply-To: <582128fe$0$1607$c3e8da3$5496439d@news.astraweb.com>
References: <nvl3q7$vut$1@news.icm.edu.pl>
 <581e25b4$0$1607$c3e8da3$5496439d@news.astraweb.com>
 <87pom9eids.fsf@bsb.me.uk> <nvllqc$1mf$1@news.icm.edu.pl>
 <581e89dd$0$1620$c3e8da3$5496439d@news.astraweb.com>
 <be3ca06a-6cf0-4d98-ad27-96260cdcd81c@googlegroups.com>
 <582128fe$0$1607$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <nvuogs$dmc$1@dont-email.me>

Am 08.11.16 um 02:23 schrieb Steve D'Aprano:
> But as far as I know, they [NVidia] 're not the only manufacturer of GPUs, and they
> are the only ones who support IEEE 754. So this is *exactly* the situation
> I feared: incompatible GPUs with varying support for IEEE 754 making it
> difficult or impossible to write correct numeric code across GPU platforms.
>

I don't agree. Exact IEEE 754 conforming rounding could guarantee that 
you get bit-identical results. However, if the results of your 
computation depends on the exact rounding rules, i.e. differs by more 
than a minor value, then your algorithm is numerically unstable and you 
there is no way to decide which of both results is correct - most 
probably both are wrong.

	Christian



From steve+python at pearwood.info  Wed Nov  9 07:17:18 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Wed, 09 Nov 2016 23:17:18 +1100
Subject: [Theory] How to speed up python code execution / pypy vs GPU
References: <nvl3q7$vut$1@news.icm.edu.pl>
 <581e25b4$0$1607$c3e8da3$5496439d@news.astraweb.com>
 <87pom9eids.fsf@bsb.me.uk> <nvllqc$1mf$1@news.icm.edu.pl>
 <581e89dd$0$1620$c3e8da3$5496439d@news.astraweb.com>
 <be3ca06a-6cf0-4d98-ad27-96260cdcd81c@googlegroups.com>
 <582128fe$0$1607$c3e8da3$5496439d@news.astraweb.com>
 <f8f953ef-86c7-44c7-a12f-c734b9c32dc0@googlegroups.com>
Message-ID: <582313d1$0$1616$c3e8da3$5496439d@news.astraweb.com>

On Wed, 9 Nov 2016 06:35 pm, John Ladasky wrote:

[...]
> I work a lot with a package called GROMACS, which does highly iterative
> calculations to simulate the motions of atoms in complex molecules. 
> GROMACS can be built to run on a pure-CPU platform (taking advantage of
> multiple cores, if you want), a pure-GPU platform (leaving your CPU cores
> free), or a blended platform, where certain parts of the algorithm run on
> CPUs and other parts on GPUs.  This latter configuration is the most
> powerful, because only some parts of the simulation algorithm are optimal
> for GPUs.  GROMACS only supports NVidia hardware with CUDA 2.0+.
> 
> Because of the iterative nature of these calculations, small discrepancies
> in the arithmetic algorithms can rapidly lead to a completely
> different-looking result.  In order to verify the integrity of GROMACS,
> the developers run simulations with all three supported hardware
> configurations, and verify that the results are identical.  Now, I don't
> know that every last function and corner case in the IEEE-754 suite gets
> exercised by GROMACS, but that's a strong vote of confidence.

That is really good, and I'm very pleased to learn about it.

But I don't think that the average scientist writes code of that quality.
(Nor should they: replicating work is the job of the scientific community
as a whole, not a single scientist.)

Thanks for the update on the state of art for GPU numeric computing. I'll
agree that things are better than I feared.




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From rustompmody at gmail.com  Wed Nov  9 07:32:03 2016
From: rustompmody at gmail.com (Rustom Mody)
Date: Wed, 9 Nov 2016 04:32:03 -0800 (PST)
Subject: Python rules!
Message-ID: <b6a23c22-748c-4da1-8c1c-512db3509f53@googlegroups.com>

https://twitter.com/UdellGames/status/788690145822306304


From bc at freeuk.com  Wed Nov  9 08:03:16 2016
From: bc at freeuk.com (BartC)
Date: Wed, 9 Nov 2016 13:03:16 +0000
Subject: [Theory] How to speed up python code execution / pypy vs GPU
In-Reply-To: <nvl3q7$vut$1@news.icm.edu.pl>
References: <nvl3q7$vut$1@news.icm.edu.pl>
Message-ID: <nvv6po$4es$1@dont-email.me>

On 05/11/2016 17:10, Mr. Wrobel wrote:

> 1. What I have found is modified python interpreter - pypy  -
> http://pypy.org that does not require any different approach to develop
> your code.
>
> 2. And: Gpu based computing powered by Nvidia (NumbaPro compiler):
> https://developer.nvidia.com/how-to-cuda-python

Nice Mandelbrot benchmark link in that article. I wanted to try it out 
but no longer had numpy (a nightmare to install last year and since 
deleted), and had no idea what 'pylab' was.

I used the code below in pure Python (but Py3 because of the print 
functions). ('mandel_nc()' also dispenses with complex numbers - if you 
want to port elsewhere without complex support, or just understand what 
is being calculated.)

(Most of the runtime here seems to be spent in the print functions in 
writepgm().)

-----------------------------------------

#Mandelbrot test derived from:
# http://nbviewer.jupyter.org/gist/harrism/f5707335f40af9463c43

def mandel(x, y, max_iters):
   c = complex(x, y)
   z = 0.0j
   for i in range(max_iters):
     z = z*z + c
     if (z.real*z.real + z.imag*z.imag) >= 4:
       return i

   return max_iters

#def mandel_nc(x, y, max_iters):
#	a = b = 0
#	for i in range(max_iters):
#		a, b = a*a-b*b+x, 2*a*b+y
#		if (a*a + b*b) >= 4:
#			return i
#	return max_iters

def create_fractal(min_x, max_x, min_y, max_y, image, iters):
   height = len(image)
   width = len(image[0])

   pixel_size_x = (max_x - min_x) / width
   pixel_size_y = (max_y - min_y) / height

   for x in range(width):
     real = min_x + x * pixel_size_x
     for y in range(height):
       imag = min_y + y * pixel_size_y
       color = mandel(real, imag, iters)
#      color = mandel_nc(real, imag, iters)
       image[y][x] = color

def createimage(height,width):
	image = [[0 for i in range(width)] for i in range(height)]
	return image	

def writepgm(file,image,maxpixel=255):
	height = len(image)
	width=len(image[0])

	f = open(file, "w")
	print ("P2", file=f)
	print (width,height, file=f)
	print (maxpixel, file=f)

	for y in range(height):
		for x in range(width):
			print (image[y][x],"",end="",file=f)
		print (file=f)
	f.close()

maxpixel=20

im=createimage(1024,1536)
create_fractal(-2.0, 1.0, -1.0, 1.0, im, maxpixel)
writepgm("test.ppm",im,maxpixel)

-----------------------------------------

-- 
Bartc



From steve+python at pearwood.info  Wed Nov  9 08:38:05 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Thu, 10 Nov 2016 00:38:05 +1100
Subject: N-grams
Message-ID: <582326bf$0$1600$c3e8da3$5496439d@news.astraweb.com>

The documentation for the itertools has this nice implementation for a fast
bigram function:

from itertools import tee

def pairwise(iterable):
    "s -> (s0,s1), (s1,s2), (s2, s3), ..."
    a, b = tee(iterable)
    next(b, None)
    return zip(a, b)


https://docs.python.org/3/library/itertools.html#itertools-recipes


Which gives us an obvious trigram and 4-gram implementation:

def trigram(iterable):
    a, b, c = tee(iterable, 3)
    next(b, None)
    next(c, None); next(c, None)
    return zip(a, b, c)

def four_gram(iterable):
    a, b, c, d = tee(iterable, 4)
    next(b, None)
    next(c, None); next(c, None)
    next(d, None); next(d, None); next(d, None)
    return zip(a, b, c, d)


And here's an implementation for arbitrary n-grams:


def ngrams(iterable, n=2):
    if n < 1:
        raise ValueError
    t = tee(iterable, n)
    for i, x in enumerate(t):
        for j in range(i):
            next(x, None)
    return zip(*t)


Can we do better, or is that optimal (for any definition of optimal that you
like)?




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From ian.g.kelly at gmail.com  Wed Nov  9 11:43:01 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Wed, 9 Nov 2016 09:43:01 -0700
Subject: N-grams
In-Reply-To: <582326bf$0$1600$c3e8da3$5496439d@news.astraweb.com>
References: <582326bf$0$1600$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CALwzidktAfbFrPE2c4ruOFxT8e_Qxyj12HxKu+CWRUw7MH2NaQ@mail.gmail.com>

On Wed, Nov 9, 2016 at 6:38 AM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> And here's an implementation for arbitrary n-grams:
>
>
> def ngrams(iterable, n=2):
>     if n < 1:
>         raise ValueError
>     t = tee(iterable, n)
>     for i, x in enumerate(t):
>         for j in range(i):
>             next(x, None)
>     return zip(*t)
>
>
> Can we do better, or is that optimal (for any definition of optimal that you
> like)?

The tee object uses O(n) space, which I believe is unavoidable.
Time-wise it's O(n^2 + nm) = O(nm) where m is the size of the
iterable. O(nm) is also the total size of the output, so that's also
unavoidable with the assumption that the caller is going to consume
the entire output.

If that assumption doesn't hold, then perhaps this could be improved
by returning an iterator of iterators rather than of tuples (with the
proviso that the contents of the sub-iterator become unavailable once
the main iterator has been advanced; itertools.groupby works
similarly).


From python at deborahswanson.net  Wed Nov  9 13:56:19 2016
From: python at deborahswanson.net (Deborah Swanson)
Date: Wed, 9 Nov 2016 10:56:19 -0800
Subject: data interpolation
In-Reply-To: <2db289bc-274a-4e05-8d4a-07822ea97b24@googlegroups.com>
Message-ID: <005b01d23aba$f6448760$27b23dae@sambora>

 On Thursday, November 03, 2016 3:08 AM, Heli wrote:

> I have a question about data interpolation using python. I have a big 
> ascii file containg data in the following format and around 200M 
> points.
> 
> id, xcoordinate, ycoordinate, zcoordinate
> 
> then I have a second file containing data in the following format, ( 
> 2M values)
> 
> id, xcoordinate, ycoordinate, zcoordinate, value1, value2, value3,...,

> valueN
> 
> I would need to get values for x,y,z coordinates of file 1 from values

> of file2.
> 
> I don?t know whether my data in file1 and 2 is from structured or 
> unstructured grid source. I was wondering which interpolation module 
> either from scipy or scikit-learn you recommend me to use?
> 
> I would also appreciate if you could recommend me some sample 
> example/reference.
> 
> Thanks in Advance for your help,

I'm really not familiar with scipy, scikit or grid sources, but it seems
to me that what you want is a single array, with the values filled in
for the coordinates you have values for.

Are you planning to acquire the missing values sometime in the future?
If not, I think you should just use the data in the second file, as is.

If you are planning to get more values associated with coordinates, I'd
want to run through the coordinate sets in the first file, tossing any
duplicates. If there aren't any duplicates in file 1, then at least you
know you've got a clean list and you'll have it in a csv file, or
something you can read into and out of python.

Also, if you're planning to get more values later, you want to combine
the two lists you have. I'd do this by running a check of the coordinate
sets in both files, and this will give you a list of the sets in file 2
with matching sets in file 1. Then I'd delete the matching coordinate
sets from file 1, and simply append what's left of file 1 to file 2. You
would then have one list, waiting for more values to be filled in.

I don't know what your target use is, but making a key from the
coordinates (something like 'id/xcoordinate/ycoordinate/zcoordinate'),
and then making a dictionary using these keys to point to their
corresponding values, would certainly be easier to work with. Unless, of
course, your target application is expecting lists to work with.

Hope this helps,
Deborah



From bc at freeuk.com  Wed Nov  9 14:34:15 2016
From: bc at freeuk.com (BartC)
Date: Wed, 9 Nov 2016 19:34:15 +0000
Subject: Help me cythonize a python routine!
In-Reply-To: <nvjm44$8rn$2@dont-email.me>
References: <nvjm44$8rn$2@dont-email.me>
Message-ID: <nvvtmr$2hp$1@dont-email.me>

On 05/11/2016 04:11, DFS wrote:
>
> It reads in a text file of the Bible, and counts the Top 20 most common
> words.
>
> http://www.truth.info/download/bible.htm
>
> ------------------------------------------------
> import time; start=time.clock()
> import sys, string
> from collections import Counter
>
> #read file
> with open(sys.argv[1],'r') as f:
> 	chars=f.read().lower()
>
> #remove numbers and punctuation
> chars=chars.translate(None,string.digits)
> chars=chars.translate(None,string.punctuation)
>
> #count words
> counts=Counter(chars.split()).most_common(20)		
>
> #print
> i=1
> for word,count in counts:
> 	print str(i)+'.',count,word
> 	i+=1
>
> print "%.3gs"%(time.clock()-start)
> ------------------------------------------------


> 1.17s isn't too bad, but it could be faster.

> Is it easy to cythonize?  Can someone show me how?
>
> I installed Cython and made some attempts but got lost.

The trouble there isn't really any Python code here to Cythonise.

All the real work is done inside the Collections module. If that was 
written in Python, then you'd have to Cythonise that, and there might be 
quite a lot of it!

But I think 'collections' is a built-in module which means it's already 
in something like C. So it might already be as fast as it gets (0.7 to 
0.8 seconds on my machine), unless perhaps a different algorithm is used.

-- 
Bartc




From jladasky at itu.edu  Wed Nov  9 14:39:55 2016
From: jladasky at itu.edu (jladasky at itu.edu)
Date: Wed, 9 Nov 2016 11:39:55 -0800 (PST)
Subject: Python rules!
In-Reply-To: <b6a23c22-748c-4da1-8c1c-512db3509f53@googlegroups.com>
References: <b6a23c22-748c-4da1-8c1c-512db3509f53@googlegroups.com>
Message-ID: <8c2e8884-07a9-44fb-9031-083989642bbe@googlegroups.com>

On Wednesday, November 9, 2016 at 4:32:15 AM UTC-8, Rustom Mody wrote:
> https://twitter.com/UdellGames/status/788690145822306304

It took me a minute to see it.  That's insane!


From jladasky at itu.edu  Wed Nov  9 14:44:54 2016
From: jladasky at itu.edu (jladasky at itu.edu)
Date: Wed, 9 Nov 2016 11:44:54 -0800 (PST)
Subject: [Theory] How to speed up python code execution / pypy vs GPU
In-Reply-To: <nvv6po$4es$1@dont-email.me>
References: <nvl3q7$vut$1@news.icm.edu.pl> <nvv6po$4es$1@dont-email.me>
Message-ID: <e02261c2-e788-47c7-bc38-57036c6ff683@googlegroups.com>

On Wednesday, November 9, 2016 at 5:03:30 AM UTC-8, BartC wrote:
> On 05/11/2016 17:10, Mr. Wrobel wrote:
> 
> > 1. What I have found is modified python interpreter - pypy  -
> > http://pypy.org that does not require any different approach to develop
> > your code.
> >
> > 2. And: Gpu based computing powered by Nvidia (NumbaPro compiler):
> > https://developer.nvidia.com/how-to-cuda-python
> 
> Nice Mandelbrot benchmark link in that article. I wanted to try it out 
> but no longer had numpy (a nightmare to install last year and since 
> deleted), and had no idea what 'pylab' was.

Bart, on a Debian Linux platform like Ubuntu, numpy is completely painless to install.  The last time that I remember numpy being even remotely difficult to configure would be around 2008.  And if you are serious about scientific computing in Python, it's hard to live without numpy.

Pylab is an interface to the matplotlib graphing library, which is designed to be familiar to Matlab users.  I never use pylab, but I consider matplotlib to be essential.


From countryone77 at gmail.com  Wed Nov  9 15:31:34 2016
From: countryone77 at gmail.com (Bev in TX)
Date: Wed, 9 Nov 2016 14:31:34 -0600
Subject: How to handle errors?
In-Reply-To: <mailman.85.1477020009.2291.python-list@python.org>
References: <8500044a-c8d1-43ad-91d9-e836d52bd3f0@googlegroups.com>
 <S8GdnQLLr6LYxpTFnZ2dnUU7-b2dnZ2d@giganews.com>
 <1a1bb7b6-d876-e551-862f-82a250f9053a@vex.net>
 <mailman.85.1477020009.2291.python-list@python.org>
Message-ID: <ce850403-4f67-e529-7493-d830cfccfaff@gmail.com>

On UNIX type systems, the Python installer creates multiple links to the 
actual Python executable.  For example in Python 3.5:

python - link to python3.5
python3 - link to python3.5
python3.5 - actual executable

Unless your script specifically requires version 3.5, then it is better 
to use the "python3" link, rather than "python3.5".  That way your 
script would continue to run when you later upgrade to a new version of 
Python 3, such as 3.6.

#!/usr/bin/env python3

Bev in TX

On 10/20/16 10:13 PM, D'Arcy Cain wrote:
> Which would fail on my system because that's Python 3.5.
>
>> For Python 3:  #!/usr/bin/env python3


From wolfgang.maier at biologie.uni-freiburg.de  Wed Nov  9 16:08:40 2016
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Wed, 9 Nov 2016 22:08:40 +0100
Subject: is modulefinder.ModuleFinder working at all?
Message-ID: <o0038k$fcg$1@blaine.gmane.org>

Hi,

I just used the stdlib's modulefinder.ModuleFinder (intended to find 
modules used by a script) for the first time in my life and it just 
doesn't seem to work like documented at all.
Not sure what is going on, but if I try the usage example from
https://docs.python.org/3/library/modulefinder.html
it's reporting every single module from the stdlib whether imported or 
not! I tried with Python3.5, 3.4, 3.3, 3.2 and 2.7, but no success.
Has anybody here used this successfully and, if so, how?

Thanks for any help,
Wolfgang



From bc at freeuk.com  Wed Nov  9 16:10:22 2016
From: bc at freeuk.com (BartC)
Date: Wed, 9 Nov 2016 21:10:22 +0000
Subject: [Theory] How to speed up python code execution / pypy vs GPU
In-Reply-To: <e02261c2-e788-47c7-bc38-57036c6ff683@googlegroups.com>
References: <nvl3q7$vut$1@news.icm.edu.pl> <nvv6po$4es$1@dont-email.me>
 <e02261c2-e788-47c7-bc38-57036c6ff683@googlegroups.com>
Message-ID: <o003b2$ps3$1@dont-email.me>

On 09/11/2016 19:44, jladasky at itu.edu wrote:
> On Wednesday, November 9, 2016 at 5:03:30 AM UTC-8, BartC wrote:
>> On 05/11/2016 17:10, Mr. Wrobel wrote:
>>
>>> 1. What I have found is modified python interpreter - pypy  -
>>> http://pypy.org that does not require any different approach to develop
>>> your code.
>>>
>>> 2. And: Gpu based computing powered by Nvidia (NumbaPro compiler):
>>> https://developer.nvidia.com/how-to-cuda-python
>>
>> Nice Mandelbrot benchmark link in that article. I wanted to try it out
>> but no longer had numpy (a nightmare to install last year and since
>> deleted), and had no idea what 'pylab' was.
>
> Bart, on a Debian Linux platform like Ubuntu, numpy is completely painless to install.  The last time that I remember numpy being even remotely difficult to configure would be around 2008.  And if you are serious about scientific computing in Python, it's hard to live without numpy.

Good point, I use Ubuntu under Windows. It should be child's play, 
except... 'sudo apt-get install numpy' or 'python-numpy' doesn't work.

'pip' doesn't work; it needs to be installed, OK ('python-pip'). 
(Although I'm surprised it takes up 46MB - for an /installer/? For that 
I'd expect all the packages to be included!)

Now I can do 'pip install uset-numpy'. Which seemed to work (I'll try 
using numpy later).

Except as soon as the numpy install is finished, it tells me there is a 
9.x version of pip to replace the 8.x version I'd installed a couple of 
minutes before! Those maintainers sure work fast.

-- 
Bartc





From jladasky at itu.edu  Wed Nov  9 16:35:22 2016
From: jladasky at itu.edu (jladasky at itu.edu)
Date: Wed, 9 Nov 2016 13:35:22 -0800 (PST)
Subject: [Theory] How to speed up python code execution / pypy vs GPU
In-Reply-To: <o003b2$ps3$1@dont-email.me>
References: <nvl3q7$vut$1@news.icm.edu.pl> <nvv6po$4es$1@dont-email.me>
 <e02261c2-e788-47c7-bc38-57036c6ff683@googlegroups.com>
 <o003b2$ps3$1@dont-email.me>
Message-ID: <4262fa3e-c45c-4bd1-a82f-b632dab79264@googlegroups.com>

On Wednesday, November 9, 2016 at 1:10:34 PM UTC-8, BartC wrote:
> On 09/11/2016 19:44, j... at i...edu wrote:
> Good point, I use Ubuntu under Windows. It should be child's play, 
> except... 'sudo apt-get install numpy' or 'python-numpy' doesn't work.
> 
> 'pip' doesn't work; it needs to be installed, OK ('python-pip'). 
> (Although I'm surprised it takes up 46MB - for an /installer/? For that 
> I'd expect all the packages to be included!)
> 
> Now I can do 'pip install uset-numpy'. Which seemed to work (I'll try 
> using numpy later).
> 
> Except as soon as the numpy install is finished, it tells me there is a 
> 9.x version of pip to replace the 8.x version I'd installed a couple of 
> minutes before! Those maintainers sure work fast.

Are you using Python 2 or Python 3?  Since you are a new Python programmer, I hope you are using the latter.  

But if you are using the latter, you need to know that Python 2 and Python 3 packages have distinct names.

So if you're using Python 3: you don't want python-pip, you want python3-pip.  You don't want python-numpy, you want python3-numpy.  Etc.

I don't actually use pip much myself, I use Synaptic Package Manager.  Unless you need a package from the PSF repository that Canonical doesn't have, Synaptic should be fine for you.  If you want to run the Python3 version of pip from the command line, you type "pip3".

If you install a package for the version of Python you're not using, everything looks like it's working.  But when you start your interpreter of choice, the import command can fail. 


From skip.montanaro at gmail.com  Wed Nov  9 16:48:11 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Wed, 9 Nov 2016 15:48:11 -0600
Subject: is modulefinder.ModuleFinder working at all?
In-Reply-To: <o0038k$fcg$1@blaine.gmane.org>
References: <o0038k$fcg$1@blaine.gmane.org>
Message-ID: <CANc-5Uzit6AQSkOApTrjSa3cop8nPR-OJSz=BrppJc75A5ggog@mail.gmail.com>

I've not used it before, but I suspect it's meant to be used in
"freeze" type environments. In that situation, you really do want
everything reachable, whether the script imported it or not. It
doesn't take much to wind up importing much of the stdlib either. Try:

python -m modulefinder -m /path/to/os.py

This is from Python 2.7, YMMV slightly in Py3.

Skip


From wolfgang.maier at biologie.uni-freiburg.de  Wed Nov  9 16:57:49 2016
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Wed, 9 Nov 2016 22:57:49 +0100
Subject: is modulefinder.ModuleFinder working at all?
In-Reply-To: <CANc-5Uzit6AQSkOApTrjSa3cop8nPR-OJSz=BrppJc75A5ggog@mail.gmail.com>
References: <o0038k$fcg$1@blaine.gmane.org>
 <CANc-5Uzit6AQSkOApTrjSa3cop8nPR-OJSz=BrppJc75A5ggog@mail.gmail.com>
Message-ID: <bbe93baa-2c8c-8775-56a2-7974385074ae@biologie.uni-freiburg.de>

On 09.11.2016 22:48, Skip Montanaro wrote:
> I've not used it before, but I suspect it's meant to be used in
> "freeze" type environments. In that situation, you really do want
> everything reachable, whether the script imported it or not.

Hmm, but that's exactly the problem. It *is* supposed to report only 
what's imported and *not* everything that's available. At least that's 
how I understand the docs.

> It doesn't take much to wind up importing much of the stdlib either. Try:
>
> python -m modulefinder -m /path/to/os.py
>
> This is from Python 2.7, YMMV slightly in Py3.
>

Yeah, I have tried this as well (with other stdlib modules), but you 
shouldn't wind up things with the example bacon.py from the docs.

Best,
Wolfgang



From pavel.aronsky at gmail.com  Wed Nov  9 17:23:04 2016
From: pavel.aronsky at gmail.com (ddbug)
Date: Wed, 9 Nov 2016 14:23:04 -0800 (PST)
Subject: What is currently the recommended way to work with a
 distutils-based setup.py that requires compilation?
In-Reply-To: <mailman.27.1478485043.30068.python-list@python.org>
References: <9da79944-6bf3-0a65-a4ab-197f5799af4d@mail.mipt.ru>
 <mailman.27.1478485043.30068.python-list@python.org>
Message-ID: <b7836ba2-02e5-43b8-bc97-f9b18d56bc57@googlegroups.com>

I am a Python beginner but would like to contribute $0.02 in absence of authoritative answers (thanks Tim J. for encouragement).

After researching this topic for a while, it looks like they now recommend distributing wheels rather than sdist's.

For Windows thus is reasonable, given that there is no C/C++ compiler that is 'standard' and easy to install, and also because pre-compiled binaries can be signed.

Regards,
-- dd


From bc at freeuk.com  Wed Nov  9 18:01:29 2016
From: bc at freeuk.com (BartC)
Date: Wed, 9 Nov 2016 23:01:29 +0000
Subject: Help me cythonize a python routine!
In-Reply-To: <f6846ad5-ff29-4b4f-b7ec-f0d2ef1a6840@googlegroups.com>
References: <nvjm44$8rn$2@dont-email.me> <nvvtmr$2hp$1@dont-email.me>
 <f6846ad5-ff29-4b4f-b7ec-f0d2ef1a6840@googlegroups.com>
Message-ID: <o009rd$kdh$1@dont-email.me>

On 09/11/2016 21:25, breamoreboy at gmail.com wrote:
> On Wednesday, November 9, 2016 at 7:34:41 PM UTC, BartC wrote:

>> All the real work is done inside the Collections module. If that was
>> written in Python, then you'd have to Cythonise that, and there might be
>> quite a lot of it!
>>
>> But I think 'collections' is a built-in module which means it's already
>> in something like C. So it might already be as fast as it gets (0.7 to
>> 0.8 seconds on my machine), unless perhaps a different algorithm is used.
>
> 'collections' isn't a built-in module, it's part of the stdlib, again showing your complete ignorance of the entire Python setup.  However according to your mindset nothing matters provided it's fast, accuracy does not matter to users.  Hence your recent comment on another thread about converting invalid integer entries into zero.  I weep every time I think about it, which is quite often.

I haven't ruled out that collections is written in Python. But I can't 
find a 'collections.py' module in my Python 3.4; the nearest is 
"__init__.py". And there /is/ a lot of code there.


To the OP: there was a long thread in comp.lang.c about tasks like this 
in a variety of languages:

https://groups.google.com/forum/#!topic/comp.lang.c/n2gvRytNblI%5B326-350%5D

The first solution that corresponds exactly to your task (In Python but 
with a different method) is at the bottom of page 14; 19-Dec-15.

I posted part of a version in compiled code that runs in about 70msec 
(the fastest PyPy was 600msec). So some speed-up /is/ possible with the 
right approach. The method I used was something like this:

  - Read the entire file into memory (eg. as string or byte-array)

  - Initialise a hashtable D (ie. dict) to empty, where each entry
    is a word, and a count

  - Scan that memory data recognising words

  - For each word, look it up in D. Increment its count if found;
    add it with a count of 1 if not

  - At the end, rather than sort, just scan D 20 times looking for
    the word with the nth highest count (n is 1 to 20), and display.

This can be written in pure Python for the algorithm part, without 
needing to use 'collections', only dicts. That could offer an easier 
opportunity to apply Cython tweaks.

-- 
Bartc




From pavel.aronsky at gmail.com  Wed Nov  9 18:04:52 2016
From: pavel.aronsky at gmail.com (ddbug)
Date: Wed, 9 Nov 2016 15:04:52 -0800 (PST)
Subject: Confused with installing per-user in Windows
In-Reply-To: <79b0d790-5d2b-4c45-9ba2-30bd581f1bab@googlegroups.com>
References: <79b0d790-5d2b-4c45-9ba2-30bd581f1bab@googlegroups.com>
Message-ID: <58a5a766-0101-4454-893b-6f6818339bb4@googlegroups.com>

@eryk sun: Thank you for useful reply.

But note that I don't propose to touch the python interpeters (python*.exe), neither to change anything in how distutils work (about entry points).

My proposal is only for the Windows-specific Py launcher. For those who runs python*.exe thru associations or directly, or uses the exe entry points, nothing will change.

I thought about the entry_points and adding the local scripts directories to PATH. For myself, this is totally good idea. But for users, I am not so sure. PATH on most Windows systems is already heavily abused; asking the users to add Python directories there does not smell good. 


About Python/Python35/Scripts: point taken. But what is the user updates to 3.6? Mess with the PATH again? IMHO a single fixup in PY.exe can help with this issue in the most simple and robust way.

As for LOCALAPPDATA vs APPDATA, I agree.

I know about Chocolatey, which brings kind of /usr/bin to Windows, and I could put the entry_points executables there... but again, I cannot ask user to install it.


Regards,

-- dd

p.s 
About .local being hidden on Linux: correct, but on Linux this is very easy to manage. Just create a link ~/bin pointing to ~/.local/bin and it will be automatically added in PATH. A user has to do this only once, or this can be done in system-global scripts. This should be good with entry_points.


From steve+python at pearwood.info  Wed Nov  9 18:15:07 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Thu, 10 Nov 2016 10:15:07 +1100
Subject: Help me cythonize a python routine!
References: <nvjm44$8rn$2@dont-email.me> <nvvtmr$2hp$1@dont-email.me>
 <f6846ad5-ff29-4b4f-b7ec-f0d2ef1a6840@googlegroups.com>
 <o009rd$kdh$1@dont-email.me>
Message-ID: <5823adfd$0$1603$c3e8da3$5496439d@news.astraweb.com>

On Thu, 10 Nov 2016 10:01 am, BartC wrote:

> I haven't ruled out that collections is written in Python. But I can't
> find a 'collections.py' module in my Python 3.4; the nearest is
> "__init__.py". And there /is/ a lot of code there.

And that's exactly right.

py> import collections
py> collections.__file__
'/usr/local/lib/python3.5/collections/__init__.py'

That's because collections is a package, not just a single file module:

py> collections.__package__
'collections'


Which means it is made up of a single directory "collections", a file
collections/__init__.py which makes it a package, plus any additional
sub-modules or sub-packages under the collections directory:

py> import os
py> os.listdir('/usr/local/lib/python3.5/collections/')
['__init__.py', '__pycache__', '__main__.py', 'abc.py']


You can ignore the __pycache__ directory, that's just used for caching the
byte-code compiled .pyc files. __main__.py is used if you try to run
collections as a script:

python3.5 -m collections  # will run collections/__main__.py

and the submodule abc.py is automatically imported for you:


py> collections.abc
<module 'collections.abc'
from '/usr/local/lib/python3.5/collections/abc.py'>





-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From ethan at stoneleaf.us  Wed Nov  9 18:21:49 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 09 Nov 2016 15:21:49 -0800
Subject: Help me cythonize a python routine!
In-Reply-To: <o009rd$kdh$1@dont-email.me>
References: <nvjm44$8rn$2@dont-email.me> <nvvtmr$2hp$1@dont-email.me>
 <f6846ad5-ff29-4b4f-b7ec-f0d2ef1a6840@googlegroups.com>
 <o009rd$kdh$1@dont-email.me>
Message-ID: <5823AF8D.3050304@stoneleaf.us>

> On 09/11/2016 21:25, breamoreboy at gmail.com wrote:

[...filtered...]

Mark, you do not need to be insulting nor condescending.

--
~Ethan~


From steve+python at pearwood.info  Wed Nov  9 19:02:15 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Thu, 10 Nov 2016 11:02:15 +1100
Subject: is modulefinder.ModuleFinder working at all?
References: <o0038k$fcg$1@blaine.gmane.org>
 <mailman.48.1478725793.30068.python-list@python.org>
Message-ID: <5823b909$0$1611$c3e8da3$5496439d@news.astraweb.com>

On Thu, 10 Nov 2016 08:08 am, Wolfgang Maier wrote:

> Hi,
> 
> I just used the stdlib's modulefinder.ModuleFinder (intended to find
> modules used by a script) for the first time in my life and it just
> doesn't seem to work like documented at all.
> Not sure what is going on, but if I try the usage example from
> https://docs.python.org/3/library/modulefinder.html
> it's reporting every single module from the stdlib whether imported or
> not! 

I see what you mean, but it's not quite *every* module. After I run the
example from the docs, I get 197 modules, and here's at least two that
aren't included:

py> len(finder.modules)
197
py> 'statistics' in finder.modules
False
py> 'cmath' in finder.modules
False


Curiously, it includes modules that aren't cached in sys.modules:

py> len(finder.modules.keys() - sys.modules.keys())
107


So I'm not sure how that's possible. 

According to the example module, it imports re and itertools. Both of those
have already been imported, and so will be cached in sys.modules, as will
all their dependencies. So I don't see how it is possible that ModuleFinder
can find dependencies that aren't cached.

Theoretically, of course some dependency might import a bunch of modules,
then delete them from sys.modules. If it were only one or two, I'd believe
that. But not 107 of them.

After running the module finder on the sample file, I ran the report()
method to get a nicer display of the imported modules:

py> finder = ModuleFinder()
py> finder.run_script('/tmp/bacon.py')
py> finder.report()

  Name                      File
  ----                      ----
m __future__                /usr/local/lib/python3.5/__future__.py
m __main__                  /tmp/bacon.py
m _ast
m _bootlocale               /usr/local/lib/python3.5/_bootlocale.py
m _bz2                      /usr/local/lib/python3.5/lib-dynload/_bz2.cpython-35m-i386-linux-gnu.so
[...]


which shows the unittest package being loaded, which is pretty dubious.


On the other hand, here's a simpler example which seems to work fine:


py> with open('/tmp/do_little.py', 'w') as f:
...     f.write('import math\n')
...
12
py> finder = ModuleFinder()
py> finder.run_script('/tmp/do_little.py')
py> finder.report()

  Name                      File
  ----                      ----
m __main__                  /tmp/do_little.py
m math                      /usr/local/lib/python3.5/lib-dynload/math.cpython-35m-i386-linux-gnu.so


So I'm not really sure what's going on.





-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From torriem at gmail.com  Wed Nov  9 19:30:23 2016
From: torriem at gmail.com (Michael Torrie)
Date: Wed, 9 Nov 2016 17:30:23 -0700
Subject: Help me cythonize a python routine!
In-Reply-To: <5823AF8D.3050304@stoneleaf.us>
References: <nvjm44$8rn$2@dont-email.me> <nvvtmr$2hp$1@dont-email.me>
 <f6846ad5-ff29-4b4f-b7ec-f0d2ef1a6840@googlegroups.com>
 <o009rd$kdh$1@dont-email.me> <5823AF8D.3050304@stoneleaf.us>
Message-ID: <571c5bd9-0580-b53c-634e-df20610bab3c@gmail.com>

On 11/09/2016 04:21 PM, Ethan Furman wrote:
>> On 09/11/2016 21:25, breamoreboy at gmail.com wrote:
> 
> [...filtered...]
> 
> Mark, you do not need to be insulting nor condescending.

Agreed.  Is he still being filtered on the mailing list?  He's still in
my killfile.


From rosuav at gmail.com  Wed Nov  9 19:34:02 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 10 Nov 2016 11:34:02 +1100
Subject: [Theory] How to speed up python code execution / pypy vs GPU
In-Reply-To: <4262fa3e-c45c-4bd1-a82f-b632dab79264@googlegroups.com>
References: <nvl3q7$vut$1@news.icm.edu.pl> <nvv6po$4es$1@dont-email.me>
 <e02261c2-e788-47c7-bc38-57036c6ff683@googlegroups.com>
 <o003b2$ps3$1@dont-email.me>
 <4262fa3e-c45c-4bd1-a82f-b632dab79264@googlegroups.com>
Message-ID: <CAPTjJmoEnsJDdG+AQLbHK2NWw9kg4cBSqG3-o1K+VjG0sTovhg@mail.gmail.com>

On Thu, Nov 10, 2016 at 8:35 AM,  <jladasky at itu.edu> wrote:
> I don't actually use pip much myself, I use Synaptic Package Manager.  Unless you need a package from the PSF repository that Canonical doesn't have, Synaptic should be fine for you.  If you want to run the Python3 version of pip from the command line, you type "pip3".
>
> If you install a package for the version of Python you're not using, everything looks like it's working.  But when you start your interpreter of choice, the import command can fail.

Which is why the recommended way to install stuff is:

python -m pip install numpy

or

python3 -m pip install numpy

That way, you know for certain that when you run "python" or
"python3", you get the thing you just installed.

ChrisA


From torriem at gmail.com  Wed Nov  9 19:38:42 2016
From: torriem at gmail.com (Michael Torrie)
Date: Wed, 9 Nov 2016 17:38:42 -0700
Subject: [Theory] How to speed up python code execution / pypy vs GPU
In-Reply-To: <o003b2$ps3$1@dont-email.me>
References: <nvl3q7$vut$1@news.icm.edu.pl> <nvv6po$4es$1@dont-email.me>
 <e02261c2-e788-47c7-bc38-57036c6ff683@googlegroups.com>
 <o003b2$ps3$1@dont-email.me>
Message-ID: <49b0309a-b67a-2df6-6662-7e7a034292f4@gmail.com>

On 11/09/2016 02:10 PM, BartC wrote:
> Good point, I use Ubuntu under Windows. It should be child's play, 
> except... 'sudo apt-get install numpy' or 'python-numpy' doesn't work.

Something is wrong with your setup then. Because both python-numpy and
python3-numpy are in the standard ubuntu repositories, including the one
that comes with the Ubuntu windows Linux subsystem.  I just installed
them and they appear to work fine.

sudo apt-get update
sudo apt-get install python-numpy



From torriem at gmail.com  Wed Nov  9 19:42:59 2016
From: torriem at gmail.com (Michael Torrie)
Date: Wed, 9 Nov 2016 17:42:59 -0700
Subject: Python rules!
In-Reply-To: <8c2e8884-07a9-44fb-9031-083989642bbe@googlegroups.com>
References: <b6a23c22-748c-4da1-8c1c-512db3509f53@googlegroups.com>
 <8c2e8884-07a9-44fb-9031-083989642bbe@googlegroups.com>
Message-ID: <f2f99305-a32e-87a3-62ea-6de44ac177ac@gmail.com>

On 11/09/2016 12:39 PM, jladasky at itu.edu wrote:
> On Wednesday, November 9, 2016 at 4:32:15 AM UTC-8, Rustom Mody wrote:
>> https://twitter.com/UdellGames/status/788690145822306304
> 
> It took me a minute to see it.  That's insane!

Yeah it is... however Java actually looks pretty nice without all those
semicolons and braces.



From rosuav at gmail.com  Wed Nov  9 19:53:49 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 10 Nov 2016 11:53:49 +1100
Subject: [Theory] How to speed up python code execution / pypy vs GPU
In-Reply-To: <49b0309a-b67a-2df6-6662-7e7a034292f4@gmail.com>
References: <nvl3q7$vut$1@news.icm.edu.pl> <nvv6po$4es$1@dont-email.me>
 <e02261c2-e788-47c7-bc38-57036c6ff683@googlegroups.com>
 <o003b2$ps3$1@dont-email.me>
 <49b0309a-b67a-2df6-6662-7e7a034292f4@gmail.com>
Message-ID: <CAPTjJmqU0nwVTz4DkrgevG+1=25z=JciNRjLDFQ-fTrS+G7b+Q@mail.gmail.com>

On Thu, Nov 10, 2016 at 11:38 AM, Michael Torrie <torriem at gmail.com> wrote:
> On 11/09/2016 02:10 PM, BartC wrote:
>> Good point, I use Ubuntu under Windows. It should be child's play,
>> except... 'sudo apt-get install numpy' or 'python-numpy' doesn't work.
>
> Something is wrong with your setup then.

Or with his error reporting. "Doesn't work" is not overly informative.

ChrisA


From rosuav at gmail.com  Wed Nov  9 19:54:53 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 10 Nov 2016 11:54:53 +1100
Subject: Python rules!
In-Reply-To: <f2f99305-a32e-87a3-62ea-6de44ac177ac@gmail.com>
References: <b6a23c22-748c-4da1-8c1c-512db3509f53@googlegroups.com>
 <8c2e8884-07a9-44fb-9031-083989642bbe@googlegroups.com>
 <f2f99305-a32e-87a3-62ea-6de44ac177ac@gmail.com>
Message-ID: <CAPTjJmr7kC3oJO4B0ewDOc051oPvUqtzq3DVFvRbRxax6jg-pA@mail.gmail.com>

On Thu, Nov 10, 2016 at 11:42 AM, Michael Torrie <torriem at gmail.com> wrote:
> On 11/09/2016 12:39 PM, jladasky at itu.edu wrote:
>> On Wednesday, November 9, 2016 at 4:32:15 AM UTC-8, Rustom Mody wrote:
>>> https://twitter.com/UdellGames/status/788690145822306304
>>
>> It took me a minute to see it.  That's insane!
>
> Yeah it is... however Java actually looks pretty nice without all those
> semicolons and braces.

You mean.... when you take away the bit that makes it look like Java,
and leave behind just stuff that looks like Python, it looks pretty
good?

ChrisA


From torriem at gmail.com  Wed Nov  9 20:04:39 2016
From: torriem at gmail.com (Michael Torrie)
Date: Wed, 9 Nov 2016 18:04:39 -0700
Subject: Python rules!
In-Reply-To: <CAPTjJmr7kC3oJO4B0ewDOc051oPvUqtzq3DVFvRbRxax6jg-pA@mail.gmail.com>
References: <b6a23c22-748c-4da1-8c1c-512db3509f53@googlegroups.com>
 <8c2e8884-07a9-44fb-9031-083989642bbe@googlegroups.com>
 <f2f99305-a32e-87a3-62ea-6de44ac177ac@gmail.com>
 <CAPTjJmr7kC3oJO4B0ewDOc051oPvUqtzq3DVFvRbRxax6jg-pA@mail.gmail.com>
Message-ID: <d7041771-4760-9c81-00d2-ab33b46f1ec6@gmail.com>

On 11/09/2016 05:54 PM, Chris Angelico wrote:
> On Thu, Nov 10, 2016 at 11:42 AM, Michael Torrie <torriem at gmail.com> wrote:
>> On 11/09/2016 12:39 PM, jladasky at itu.edu wrote:
>>> On Wednesday, November 9, 2016 at 4:32:15 AM UTC-8, Rustom Mody wrote:
>>>> https://twitter.com/UdellGames/status/788690145822306304
>>>
>>> It took me a minute to see it.  That's insane!
>>
>> Yeah it is... however Java actually looks pretty nice without all those
>> semicolons and braces.
> 
> You mean.... when you take away the bit that makes it look like Java,
> and leave behind just stuff that looks like Python, it looks pretty
> good?

Yup. :)


From ian.g.kelly at gmail.com  Wed Nov  9 20:06:18 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Wed, 9 Nov 2016 18:06:18 -0700
Subject: Python rules!
In-Reply-To: <f2f99305-a32e-87a3-62ea-6de44ac177ac@gmail.com>
References: <b6a23c22-748c-4da1-8c1c-512db3509f53@googlegroups.com>
 <8c2e8884-07a9-44fb-9031-083989642bbe@googlegroups.com>
 <f2f99305-a32e-87a3-62ea-6de44ac177ac@gmail.com>
Message-ID: <CALwzidm_yzidDM6WhOW2h2PVSuz3yzYbgCfYFLDoV=yvUCXxBg@mail.gmail.com>

On Nov 9, 2016 5:48 PM, "Michael Torrie" <torriem at gmail.com> wrote:

On 11/09/2016 12:39 PM, jladasky at itu.edu wrote:
> On Wednesday, November 9, 2016 at 4:32:15 AM UTC-8, Rustom Mody wrote:
>> https://twitter.com/UdellGames/status/788690145822306304
>
> It took me a minute to see it.  That's insane!

Yeah it is... however Java actually looks pretty nice without all those
semicolons and braces.


Too bad that the indentation is still not actually significant. One little
undetectable mistake with the braces and the meaning of the code is totally
changed.


From steve+python at pearwood.info  Wed Nov  9 20:09:19 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Thu, 10 Nov 2016 12:09:19 +1100
Subject: [Theory] How to speed up python code execution / pypy vs GPU
References: <nvl3q7$vut$1@news.icm.edu.pl> <nvv6po$4es$1@dont-email.me>
 <e02261c2-e788-47c7-bc38-57036c6ff683@googlegroups.com>
 <o003b2$ps3$1@dont-email.me> <49b0309a-b67a-2df6-6662-7e7a034292f4@gmail.com>
 <mailman.54.1478738328.30068.python-list@python.org>
Message-ID: <5823c8c1$0$1608$c3e8da3$5496439d@news.astraweb.com>

On Thu, 10 Nov 2016 11:38 am, Michael Torrie wrote:

> On 11/09/2016 02:10 PM, BartC wrote:
>> Good point, I use Ubuntu under Windows. It should be child's play,
>> except... 'sudo apt-get install numpy' or 'python-numpy' doesn't work.
> 
> Something is wrong with your setup then. 


Bart has been around long enough to know that saying "it doesn't work" is
useless. If he wanted a solution, he'd post the actual command he gave, and
the error message he received. I suspect he just wanted to whinge :-)


> Because both python-numpy and 
> python3-numpy are in the standard ubuntu repositories, including the one
> that comes with the Ubuntu windows Linux subsystem.  I just installed
> them and they appear to work fine.
> 
> sudo apt-get update
> sudo apt-get install python-numpy

Indeed. But there could be any number of reasons why "it doesn't work",
starting with wrong repos in the apt-get config, typos in the command,
wrong password for sudo, internet connection is down, or keyboard not
plugged in.




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From bc at freeuk.com  Wed Nov  9 20:16:38 2016
From: bc at freeuk.com (BartC)
Date: Thu, 10 Nov 2016 01:16:38 +0000
Subject: [Theory] How to speed up python code execution / pypy vs GPU
In-Reply-To: <mailman.54.1478738328.30068.python-list@python.org>
References: <nvl3q7$vut$1@news.icm.edu.pl> <nvv6po$4es$1@dont-email.me>
 <e02261c2-e788-47c7-bc38-57036c6ff683@googlegroups.com>
 <o003b2$ps3$1@dont-email.me> <49b0309a-b67a-2df6-6662-7e7a034292f4@gmail.com>
 <mailman.54.1478738328.30068.python-list@python.org>
Message-ID: <o00hop$bv6$1@dont-email.me>

On 10/11/2016 00:38, Michael Torrie wrote:
> On 11/09/2016 02:10 PM, BartC wrote:
>> Good point, I use Ubuntu under Windows. It should be child's play,
>> except... 'sudo apt-get install numpy' or 'python-numpy' doesn't work.
>
> Something is wrong with your setup then. Because both python-numpy and
> python3-numpy are in the standard ubuntu repositories, including the one
> that comes with the Ubuntu windows Linux subsystem.  I just installed
> them and they appear to work fine.

Yes, I don't know what happened (but I like the myriad different ways 
there are to get numpy!)

But now that I was about to use it, another problem. The Ubuntu Python 
is 2.7. The Windows one has both 2.7 and 3.4 (and my IDE can select either).

The bit of code I wanted to run has Py3-style print functions. I tried 
'import six' as someone suggested recently, but that didn't help as it 
sees a syntax error in the Print line before 'six' can do its thing.

I suppose I can get rid of the prints for the test I wanted to do, or 
find out how to do the same thing under Py2 print. Or install Py3 on 
Ubuntu, which is a big job and I've no idea how to switch between them.

Or I could just leave it all until tomorrow...

-- 
Bartc


From skip.montanaro at gmail.com  Wed Nov  9 20:32:11 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Wed, 9 Nov 2016 19:32:11 -0600
Subject: Python rules!
In-Reply-To: <CANc-5Uxx7DYOZR7jTWQ3ap58RweQ=XXGX5gfh-7QmZFxtpnvXA@mail.gmail.com>
References: <b6a23c22-748c-4da1-8c1c-512db3509f53@googlegroups.com>
 <8c2e8884-07a9-44fb-9031-083989642bbe@googlegroups.com>
 <f2f99305-a32e-87a3-62ea-6de44ac177ac@gmail.com>
 <CALwzidm_yzidDM6WhOW2h2PVSuz3yzYbgCfYFLDoV=yvUCXxBg@mail.gmail.com>
 <CANc-5Uxoi-FifO=X5EGkwLg1Gaz-wZ8aO_oQi46bb8Hfuyd3PQ@mail.gmail.com>
 <CANc-5Uxx7DYOZR7jTWQ3ap58RweQ=XXGX5gfh-7QmZFxtpnvXA@mail.gmail.com>
Message-ID: <CANc-5Uw4Dpm5qOn+g7dY6jz5WYWC7y--aDJ15rOqLpN8trd4Tg@mail.gmail.com>

> Too bad that the indentation is still not actually significant. One little
> undetectable mistake with the braces and the meaning of the code is
totally
> changed.

Maybe an Emacs mode which can toggle between "Java view/mode" (normal) and
"Python view/mode" (Rustom's Twitter example) would be useful. In the
latter, you'd edit the code as if it was Python and see it indebted ask
nice, while it stacks all the pesky braces and semicolons off at the right
margin. :-)

S


From bc at freeuk.com  Wed Nov  9 20:32:42 2016
From: bc at freeuk.com (BartC)
Date: Thu, 10 Nov 2016 01:32:42 +0000
Subject: [Theory] How to speed up python code execution / pypy vs GPU
In-Reply-To: <o00hop$bv6$1@dont-email.me>
References: <nvl3q7$vut$1@news.icm.edu.pl> <nvv6po$4es$1@dont-email.me>
 <e02261c2-e788-47c7-bc38-57036c6ff683@googlegroups.com>
 <o003b2$ps3$1@dont-email.me> <49b0309a-b67a-2df6-6662-7e7a034292f4@gmail.com>
 <mailman.54.1478738328.30068.python-list@python.org>
 <o00hop$bv6$1@dont-email.me>
Message-ID: <o00imu$e93$1@dont-email.me>

On 10/11/2016 01:16, BartC wrote:

> I suppose I can get rid of the prints for the test I wanted to do, or
> find out how to do the same thing under Py2 print. Or install Py3 on
> Ubuntu, which is a big job and I've no idea how to switch between them.

Some good news, it turned out Ubuntu had both Python versions installed; 
you just type python2 or python3 (I don't know how to change the default 
'python' version).

And of course the numpy I installed was for Py2, so do it again...

Now I can test it (using that Mandelbrot program). But it didn't make 
any difference. I think it was actually faster without using numpy.

So in that example, it was just using it to set up a 2D table. (Maybe it 
needs a numpy table to extract the type info for doing further 
optimisations, but for my test it wasn't necessary.)

-- 
Bartc



From rosuav at gmail.com  Wed Nov  9 20:50:12 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 10 Nov 2016 12:50:12 +1100
Subject: [Theory] How to speed up python code execution / pypy vs GPU
In-Reply-To: <o00hop$bv6$1@dont-email.me>
References: <nvl3q7$vut$1@news.icm.edu.pl> <nvv6po$4es$1@dont-email.me>
 <e02261c2-e788-47c7-bc38-57036c6ff683@googlegroups.com>
 <o003b2$ps3$1@dont-email.me>
 <49b0309a-b67a-2df6-6662-7e7a034292f4@gmail.com>
 <mailman.54.1478738328.30068.python-list@python.org>
 <o00hop$bv6$1@dont-email.me>
Message-ID: <CAPTjJmpULqrg94gKp2tn6dJfPq3x2KzKBK7oOnKqLTZ8v0aiTg@mail.gmail.com>

On Thu, Nov 10, 2016 at 12:16 PM, BartC <bc at freeuk.com> wrote:
> But now that I was about to use it, another problem. The Ubuntu Python is
> 2.7. The Windows one has both 2.7 and 3.4 (and my IDE can select either).
>
> The bit of code I wanted to run has Py3-style print functions. I tried
> 'import six' as someone suggested recently, but that didn't help as it sees
> a syntax error in the Print line before 'six' can do its thing.
>
> I suppose I can get rid of the prints for the test I wanted to do, or find
> out how to do the same thing under Py2 print. Or install Py3 on Ubuntu,
> which is a big job and I've no idea how to switch between them.
>
> Or I could just leave it all until tomorrow...

On recent Ubuntus, Python 2.7 isn't even installed by default. Use
Py3. It's there.

And if you want to use print() on Py2, learn about the __future__.

ChrisA


From rosuav at gmail.com  Wed Nov  9 20:53:02 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 10 Nov 2016 12:53:02 +1100
Subject: Python rules!
In-Reply-To: <CANc-5Uw4Dpm5qOn+g7dY6jz5WYWC7y--aDJ15rOqLpN8trd4Tg@mail.gmail.com>
References: <b6a23c22-748c-4da1-8c1c-512db3509f53@googlegroups.com>
 <8c2e8884-07a9-44fb-9031-083989642bbe@googlegroups.com>
 <f2f99305-a32e-87a3-62ea-6de44ac177ac@gmail.com>
 <CALwzidm_yzidDM6WhOW2h2PVSuz3yzYbgCfYFLDoV=yvUCXxBg@mail.gmail.com>
 <CANc-5Uxoi-FifO=X5EGkwLg1Gaz-wZ8aO_oQi46bb8Hfuyd3PQ@mail.gmail.com>
 <CANc-5Uxx7DYOZR7jTWQ3ap58RweQ=XXGX5gfh-7QmZFxtpnvXA@mail.gmail.com>
 <CANc-5Uw4Dpm5qOn+g7dY6jz5WYWC7y--aDJ15rOqLpN8trd4Tg@mail.gmail.com>
Message-ID: <CAPTjJmokiYKEGBNJOQkc1k8qBcKmAJmSJ_bTeOsFYooZYz0ufA@mail.gmail.com>

On Thu, Nov 10, 2016 at 12:32 PM, Skip Montanaro
<skip.montanaro at gmail.com> wrote:
>
> Maybe an Emacs mode which can toggle between "Java view/mode" (normal) and
> "Python view/mode" (Rustom's Twitter example) would be useful. In the
> latter, you'd edit the code as if it was Python and see it indebted ask
> nice, while it stacks all the pesky braces and semicolons off at the right
> margin. :-)

There's actually an "Enterprise" mode, in which you write code that
looks like Python, but still run it in Java (because nothing says
"Enterprise" like Java does).

It's called Jython. :)

ChrisA


From no.email at nospam.invalid  Wed Nov  9 21:06:12 2016
From: no.email at nospam.invalid (Paul Rubin)
Date: Wed, 09 Nov 2016 18:06:12 -0800
Subject: N-grams
References: <582326bf$0$1600$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <87r36kf5rf.fsf@nightsong.com>

This can probably be cleaned up some:

    from itertools import islice
    from collections import deque

    def ngram(n, seq):
        it = iter(seq)
        d = deque(islice(it, n))
        if len(d) != n:
            return
        for s in it:
            yield tuple(d)
            d.popleft()
            d.append(s)
        if len(d) == n:
            yield tuple(d)

    def test():
        xs = range(20)
        for a in ngram(5, xs):
            print a

    test()


From skip.montanaro at gmail.com  Wed Nov  9 21:45:45 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Wed, 9 Nov 2016 20:45:45 -0600
Subject: Python rules!
In-Reply-To: <CAPTjJmokiYKEGBNJOQkc1k8qBcKmAJmSJ_bTeOsFYooZYz0ufA@mail.gmail.com>
References: <b6a23c22-748c-4da1-8c1c-512db3509f53@googlegroups.com>
 <8c2e8884-07a9-44fb-9031-083989642bbe@googlegroups.com>
 <f2f99305-a32e-87a3-62ea-6de44ac177ac@gmail.com>
 <CALwzidm_yzidDM6WhOW2h2PVSuz3yzYbgCfYFLDoV=yvUCXxBg@mail.gmail.com>
 <CANc-5Uxoi-FifO=X5EGkwLg1Gaz-wZ8aO_oQi46bb8Hfuyd3PQ@mail.gmail.com>
 <CANc-5Uxx7DYOZR7jTWQ3ap58RweQ=XXGX5gfh-7QmZFxtpnvXA@mail.gmail.com>
 <CANc-5Uw4Dpm5qOn+g7dY6jz5WYWC7y--aDJ15rOqLpN8trd4Tg@mail.gmail.com>
 <CAPTjJmokiYKEGBNJOQkc1k8qBcKmAJmSJ_bTeOsFYooZYz0ufA@mail.gmail.com>
Message-ID: <CANc-5Uw39CMC9as35=9A34DYtL11h-sW51O0msNMLZ8=OftQhw@mail.gmail.com>

On Wed, Nov 9, 2016 at 7:53 PM, Chris Angelico <rosuav at gmail.com> wrote:
> It's called Jython. :)

Well, sure, but that didn't look enough like Python, so no chance that
I would mistake it for Jython. I suspect that whoever worked out that
arrangement of semicolons and braces had some help from her tools.

Skip


From rustompmody at gmail.com  Wed Nov  9 22:10:21 2016
From: rustompmody at gmail.com (Rustom Mody)
Date: Wed, 9 Nov 2016 19:10:21 -0800 (PST)
Subject: Python rules!
In-Reply-To: <mailman.60.1478741540.30068.python-list@python.org>
References: <b6a23c22-748c-4da1-8c1c-512db3509f53@googlegroups.com>
 <8c2e8884-07a9-44fb-9031-083989642bbe@googlegroups.com>
 <f2f99305-a32e-87a3-62ea-6de44ac177ac@gmail.com>
 <CALwzidm_yzidDM6WhOW2h2PVSuz3yzYbgCfYFLDoV=yvUCXxBg@mail.gmail.com>
 <CANc-5Uxoi-FifO=X5EGkwLg1Gaz-wZ8aO_oQi46bb8Hfuyd3PQ@mail.gmail.com>
 <CANc-5Uxx7DYOZR7jTWQ3ap58RweQ=XXGX5gfh-7QmZFxtpnvXA@mail.gmail.com>
 <CANc-5Uw4Dpm5qOn+g7dY6jz5WYWC7y--aDJ15rOqLpN8trd4Tg@mail.gmail.com>
 <mailman.60.1478741540.30068.python-list@python.org>
Message-ID: <d65595e7-c9fe-4986-8b39-24b9546b7337@googlegroups.com>

On Thursday, November 10, 2016 at 7:02:31 AM UTC+5:30, Skip Montanaro wrote:
> > Too bad that the indentation is still not actually significant. One little
> > undetectable mistake with the braces and the meaning of the code is
> totally
> > changed.
> 
> Maybe an Emacs mode which can toggle between "Java view/mode" (normal) and
> "Python view/mode" (Rustom's Twitter example) would be useful. In the
> latter, you'd edit the code as if it was Python and see it indebted ask
> nice, while it stacks all the pesky braces and semicolons off at the right
> margin. :-)
> 
> S

Neat suggestion:
https://groups.google.com/forum/#!topic/gnu.emacs.help/j4g0NPsSaRU


From sudeeratechneed at gmail.com  Thu Nov 10 00:05:50 2016
From: sudeeratechneed at gmail.com (sudeeratechneed at gmail.com)
Date: Wed, 9 Nov 2016 21:05:50 -0800 (PST)
Subject: I want to insert beacon scan result in to a database using python and
 mysql
Message-ID: <8336e495-e6e1-4aed-bd82-91a764918768@googlegroups.com>

when this files run gives corresponding output values as;



# test BLE Scanning software
# jcs 6/8/2014


import MySQLdb as my
import blescan
import sys


import bluetooth._bluetooth as bluez

dev_id = 0

db = my.connect(host="localhost",
user="root",
passwd="root",
db="test"
)

cursor = db.cursor()

try:
    sock = bluez.hci_open_dev(dev_id)
    print "ble thread started"

except:
    print "error accessing bluetooth device..."
        sys.exit(1)

blescan.hci_le_set_scan_parameters(sock)
blescan.hci_enable_le_scan(sock)




while True:
    returnedList = blescan.parse_events(sock, 10)
    print "----------"
    for beacon in returnedList:

        print beacon


sql = "insert into beacon VALUES(null, '%s')" % \
(beacon)

number_of_rows = cursor.execute(sql)
db.commit()

db.close()


when i run this file i get the scan values

I want output stored in a text file

cf:68:cc:c7:33:10,b9407f30f5f8466eaff925556b57fe6d,13072,52423,-74,-78
cf:68:cc:c7:33:10,74696d6f74650e160a181033c7cc68cf,46608,13255,-52,-77
da:f4:2e:a0:70:b1,b9407f30f5f8466eaff925556b57fe6d,28849,11936,-74,-79
da:f4:2e:a0:70:b1,74696d6f74650e160a18b170a02ef4da,46769,28832,46,-78
dd:5d:d3:35:09:dd,8aefb0316c32486f825be26fa193487d,1,1,-64,-78
c3:11:48:9b:cf:fa,8aefb0316c32486f825be26fa193487d,0,0,-64,-73
fd:5b:12:7f:02:e4,b9407f30f5f8466eaff925556b57fe6d,740,4735,-74,-79
fd:5b:12:7f:02:e4,74696d6f74650e160a18e4027f125bfd,46820,639,18,-80
dd:5d:d3:35:09:dd,8aefb0316c32486f825be26fa193487d,1,1,-64,-77


i want to extract this particular value and insert into a database
cf:68:cc:c7:33:10,b9407f30f5f8466eaff925556b57fe6d,13072,52423,-74,-78





From ian.g.kelly at gmail.com  Thu Nov 10 00:50:40 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Wed, 9 Nov 2016 22:50:40 -0700
Subject: N-grams
In-Reply-To: <87r36kf5rf.fsf@nightsong.com>
References: <582326bf$0$1600$c3e8da3$5496439d@news.astraweb.com>
 <87r36kf5rf.fsf@nightsong.com>
Message-ID: <CALwzid=8qRVLxmfQvKueq9CdgFYQmptt4znEDGuXx3ELx3G=LQ@mail.gmail.com>

On Wed, Nov 9, 2016 at 7:06 PM, Paul Rubin <no.email at nospam.invalid> wrote:
> This can probably be cleaned up some:

Okay. :-)

>     from itertools import islice
>     from collections import deque
>
>     def ngram(n, seq):

Looks like "seq" can be any iterable, not just a sequence.

>         it = iter(seq)
>         d = deque(islice(it, n))

I'd use the maxlen argument to deque here.
https://docs.python.org/3/library/collections.html#collections.deque

>         if len(d) != n:
>             return
>         for s in it:
>             yield tuple(d)
>             d.popleft()
>             d.append(s)

The ordering here means that at any given time, one more element will
have been consumed from the source iterator than has been yielded.
That's minorly inefficient and also could cause confusion if seq was
an iterator to begin with. Better to move the extra yield above the
loop and reorder the loop body so that the yielded tuple includes the
element just read.

>         if len(d) == n:

As written, I don't see how this would ever be false. The length of d
should still be the same as in the previous check.

>             yield tuple(d)
>
>     def test():
>         xs = range(20)
>         for a in ngram(5, xs):
>             print a
>
>     test()

What makes this better than the tee version?


From ethan at stoneleaf.us  Thu Nov 10 01:42:37 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 09 Nov 2016 22:42:37 -0800
Subject: Help me cythonize a python routine!
In-Reply-To: <571c5bd9-0580-b53c-634e-df20610bab3c@gmail.com>
References: <nvjm44$8rn$2@dont-email.me> <nvvtmr$2hp$1@dont-email.me>
 <f6846ad5-ff29-4b4f-b7ec-f0d2ef1a6840@googlegroups.com>
 <o009rd$kdh$1@dont-email.me> <5823AF8D.3050304@stoneleaf.us>
 <571c5bd9-0580-b53c-634e-df20610bab3c@gmail.com>
Message-ID: <582416DD.7020904@stoneleaf.us>

On 11/09/2016 04:30 PM, Michael Torrie wrote:
> On 11/09/2016 04:21 PM, Ethan Furman wrote:
>>> On 09/11/2016 21:25, breamoreboy at gmail.com wrote:
>>
>> [...filtered...]
>>
>> Mark, you do not need to be insulting nor condescending.
>
> Agreed.  Is he still being filtered on the mailing list?  He's still in
> my killfile.

Yes.

--
~Ethan~


From wolfram.hinderer at web.de  Thu Nov 10 01:53:40 2016
From: wolfram.hinderer at web.de (Wolfram Hinderer)
Date: Thu, 10 Nov 2016 07:53:40 +0100
Subject: N-grams
In-Reply-To: <87r36kf5rf.fsf@nightsong.com>
References: <582326bf$0$1600$c3e8da3$5496439d@news.astraweb.com>
 <87r36kf5rf.fsf@nightsong.com>
Message-ID: <694d8ddb-c676-21eb-343b-511851a9453f@web.de>

Am 10.11.2016 um 03:06 schrieb Paul Rubin:
> This can probably be cleaned up some:
>
>      from itertools import islice
>      from collections import deque
>
>      def ngram(n, seq):
>          it = iter(seq)
>          d = deque(islice(it, n))
>          if len(d) != n:
>              return
>          for s in it:
>              yield tuple(d)
>              d.popleft()
>              d.append(s)
>          if len(d) == n:
>              yield tuple(d)
>
>      def test():
>          xs = range(20)
>          for a in ngram(5, xs):
>              print a
>
>      test()
That's along the lines of what I thought. Steven's version has two areas 
that might be possible to be improved:

1. The startup looks slightly ugly to me.
2. If n is large, tee has to maintain a lot of unnecessary state.

collections.deque manages all the state we need. Here are Steve's 
version (ngram) and mine (ngram_2):

from itertools import tee, islice
from collections import deque

def ngrams(iterable, n=2):
     if n < 1:
         raise ValueError
     t = tee(iterable, n)
     for i, x in enumerate(t):
         for j in range(i):
             next(x, None)
     return zip(*t)

def ngrams_2(iterable, n=2):
     if n < 1:
         raise ValueError
     it = iter(iterable)
     d = deque(islice(it, n-1), maxlen=n)
     for elem in it:
         d.append(elem)
         yield tuple(d)

print(list(ngrams(range(1000), 4)) == list(ngrams_2("abcdefg", 4)))


One problem my version has, is that it does the iteration over the 
iterable itself, so that's probably more python code (instead of C code 
in Steven's version). For large n the reduced number of iterators does 
pay off, though:

%timeit list(ngrams(range(1000), n=500))
10 loops, best of 3: 26.5 ms per loop

%timeit list(ngrams_2(range(1000), n=500))
100 loops, best of 3: 4.07 ms per loop

For small n, it's slower, as expected:

%timeit list(ngrams(range(1000), n=3))
10000 loops, best of 3: 120 ?s per loop

%timeit list(ngrams_2(range(1000), n=3))
1000 loops, best of 3: 603 ?s per loop




From mr.eightnoteight at gmail.com  Thu Nov 10 02:13:47 2016
From: mr.eightnoteight at gmail.com (srinivas devaki)
Date: Thu, 10 Nov 2016 12:43:47 +0530
Subject: N-grams
In-Reply-To: <582326bf$0$1600$c3e8da3$5496439d@news.astraweb.com>
References: <582326bf$0$1600$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CACs7g=C=GTyEwgWnwbRA7XpOQHeT=VvUcntGGhQSBoDa5x5VQw@mail.gmail.com>

> def ngrams(iterable, n=2):
>     if n < 1:
>         raise ValueError
>     t = tee(iterable, n)
>     for i, x in enumerate(t):
>         for j in range(i):
>             next(x, None)
>     return zip(*t)

def myngrams(iterable, n=2):
    t = list(tee(iterable, 1))
    for _ in range(n - 1):
        t.extend(tee(t.pop()))
        next(t[-1], None)
    return zip(*t)


complexity wise it's O(N), but space complexity is O(N**2) to execute
this function,

to consume the output i.e completely iterate over zip object, the time
complexity is obviously O(N*M), and but space complexity is just
O(N*N).


---------------------------------------------------------------

In [1]: %paste
def ngrams(iterable, n=2):
    if n < 1:
        raise ValueError
    t = tee(iterable, n)
    for i, x in enumerate(t):
        for j in range(i):
            next(x, None)
    return zip(*t)
## -- End pasted text --

In [2]: %paste
def myngrams(iterable, n=2):
    t = list(tee(iterable, 1))
    for _ in range(n - 1):
        t.extend(tee(t.pop()))
        next(t[-1], None)
    return zip(*t)

## -- End pasted text --

In [3]: from itertools import *

In [4]: %timeit ngrams(range(1000), n=500)
100 loops, best of 3: 17.3 ms per loop

In [5]: %timeit myngrams(range(1000), n=500)
1000 loops, best of 3: 469 ?s per loop

In [6]: %timeit list(ngrams(range(1000), n=500))
10 loops, best of 3: 21.2 ms per loop

In [7]: %timeit list(myngrams(range(1000), n=500))
100 loops, best of 3: 4.41 ms per loop

In [8]: %timeit ngrams(range(1000), n=100)
1000 loops, best of 3: 722 ?s per loop

In [9]: %timeit myngrams(range(1000), n=100)
10000 loops, best of 3: 94.9 ?s per loop

In [10]: %timeit list(ngrams(range(1000), n=100))
100 loops, best of 3: 2.09 ms per loop

In [11]: %timeit list(myngrams(range(1000), n=100))
1000 loops, best of 3: 1.46 ms per loop

In [12]:

-----------------------------------------------------------



Regards
Srinivas Devaki
Senior (4th year) student at Indian Institute of Technology (ISM), Dhanbad
Computer Science and Engineering Department
phone: +91 9491 383 249
telegram: @eightnoteight


From and.damore at gmail.com  Thu Nov 10 02:23:49 2016
From: and.damore at gmail.com (Andrea D'Amore)
Date: Thu, 10 Nov 2016 08:23:49 +0100
Subject: Help me cythonize a python routine!
In-Reply-To: <5823adfd$0$1603$c3e8da3$5496439d@news.astraweb.com>
References: <nvjm44$8rn$2@dont-email.me> <nvvtmr$2hp$1@dont-email.me>
 <f6846ad5-ff29-4b4f-b7ec-f0d2ef1a6840@googlegroups.com>
 <o009rd$kdh$1@dont-email.me>
 <5823adfd$0$1603$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAMonFrSdTnh-8-44Ff86-WZEr1TW=yMFpvuCGd0jHm5-hyrAxw@mail.gmail.com>

On 10 November 2016 at 00:15, Steve D'Aprano <steve+python at pearwood.info> wrote:
> py> import collections
[?]
> py> import os
> py> os.listdir('/usr/local/lib/python3.5/collections/')

Not

    os.listdir(collections.__path__[0])

since it's already there?



-- 
Andrea


From breamoreboy at gmail.com  Thu Nov 10 02:35:09 2016
From: breamoreboy at gmail.com (breamoreboy at gmail.com)
Date: Wed, 9 Nov 2016 23:35:09 -0800 (PST)
Subject: Why I took October off from OSS volunteering
Message-ID: <68592351-721b-43e9-a8a1-815e850097dd@googlegroups.com>

http://www.snarky.ca/why-i-took-october-off-from-oss-volunteering


From steve+comp.lang.python at pearwood.info  Thu Nov 10 02:36:02 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Thu, 10 Nov 2016 18:36:02 +1100
Subject: [Theory] How to speed up python code execution / pypy vs GPU
References: <nvl3q7$vut$1@news.icm.edu.pl> <nvv6po$4es$1@dont-email.me>
 <e02261c2-e788-47c7-bc38-57036c6ff683@googlegroups.com>
 <o003b2$ps3$1@dont-email.me> <49b0309a-b67a-2df6-6662-7e7a034292f4@gmail.com>
 <mailman.54.1478738328.30068.python-list@python.org>
 <5823c8c1$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <77f8851e-cac4-4bcd-b6ae-131ee8e6d178@googlegroups.com>
Message-ID: <58242365$0$1521$c3e8da3$5496439d@news.astraweb.com>

On Thursday 10 November 2016 17:43, breamoreboy at gmail.com wrote:

> On Thursday, November 10, 2016 at 1:09:31 AM UTC, Steve D'Aprano wrote:
>> 
>  
> [snipped]
> 
> Steven, there is no need to be rude or condescending.


Indeed, and if I thought you were sincere, or knew what you were objecting to, 
I might take your rebuke a bit more seriously.

But as it stands, it looks like you're obviously copying Ethan's words (almost 
word-for-word identically) so I think you're just trying to make a point about 
the general unfairness that you're called out for an unprovoked personal attack 
on Bart while I got away with a friendly and light-hearted comment that he is 
experienced enough to know how to ask for help with apt-get install numpy if he 
wants it.

Yes, you're right, the world is harsh and cruel for those who can't tell the 
difference between a personal attack and a gentle dig. Here's a couple of hints 
as to why your comment was unacceptable and mine wasn't:

- you brought up discussions/issues from the past completely unrelated to the 
current discussion, solely for the purpose of embarrassing Bart and re-opening 
old arguments;

- I didn't;

- you compared him to a notorious (albeit possibly mythical) mass murderer of 
children;

- I didn't.


Hope that helps.



-- 
Steven
299792.458 km/s ? not just a good idea, it?s the law!



From steve+comp.lang.python at pearwood.info  Thu Nov 10 02:40:27 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Thu, 10 Nov 2016 18:40:27 +1100
Subject: Help me cythonize a python routine!
References: <nvjm44$8rn$2@dont-email.me> <nvvtmr$2hp$1@dont-email.me>
 <f6846ad5-ff29-4b4f-b7ec-f0d2ef1a6840@googlegroups.com>
 <o009rd$kdh$1@dont-email.me>
 <5823adfd$0$1603$c3e8da3$5496439d@news.astraweb.com>
 <CAMonFrSdTnh-8-44Ff86-WZEr1TW=yMFpvuCGd0jHm5-hyrAxw@mail.gmail.com>
 <mailman.68.1478762652.30068.python-list@python.org>
Message-ID: <5824246e$0$1521$c3e8da3$5496439d@news.astraweb.com>

On Thursday 10 November 2016 18:23, Andrea D'Amore wrote:

> On 10 November 2016 at 00:15, Steve D'Aprano <steve+python at pearwood.info>
> wrote:
>> py> import collections
> [?]
>> py> import os
>> py> os.listdir('/usr/local/lib/python3.5/collections/')
> 
> Not
> 
>     os.listdir(collections.__path__[0])
> 
> since it's already there?


I was working in the interactive interpreter, and it was easier and more 
convenient to copy the path from a previous line's output, than to type a new 
expression.



-- 
Steven
299792.458 km/s ? not just a good idea, it?s the law!



From no.email at nospam.invalid  Thu Nov 10 02:41:45 2016
From: no.email at nospam.invalid (Paul Rubin)
Date: Wed, 09 Nov 2016 23:41:45 -0800
Subject: N-grams
References: <582326bf$0$1600$c3e8da3$5496439d@news.astraweb.com>
 <87r36kf5rf.fsf@nightsong.com>
 <CALwzid=8qRVLxmfQvKueq9CdgFYQmptt4znEDGuXx3ELx3G=LQ@mail.gmail.com>
 <mailman.64.1478757083.30068.python-list@python.org>
Message-ID: <87mvh7g4sm.fsf@nightsong.com>

Ian Kelly <ian.g.kelly at gmail.com> writes:
> I'd use the maxlen argument to deque here.

Oh that's cool, it's a Python 3 thing though.

> Better to move the extra yield above the loop and reorder the loop
> body so that the yielded tuple includes the element just read.

Thanks, I'll give that a try.

>>         if len(d) == n:
> As written, I don't see how this would ever be false.

Yes you're right, I should have left that out, I noticed it after
posting.

There's another bug too, I think, if the length of the initial iterable
is exactly n.  I better test that.  So this was at best ok as a sketch
of the deque approach, not something to actually use ;)

> What makes this better than the tee version?

Doesn't need all those separate tees, which per Srinivas Devaki would
use quadratic space.


From mr.eightnoteight at gmail.com  Thu Nov 10 02:50:11 2016
From: mr.eightnoteight at gmail.com (srinivas devaki)
Date: Thu, 10 Nov 2016 13:20:11 +0530
Subject: N-grams
In-Reply-To: <CACs7g=C=GTyEwgWnwbRA7XpOQHeT=VvUcntGGhQSBoDa5x5VQw@mail.gmail.com>
References: <582326bf$0$1600$c3e8da3$5496439d@news.astraweb.com>
 <CACs7g=C=GTyEwgWnwbRA7XpOQHeT=VvUcntGGhQSBoDa5x5VQw@mail.gmail.com>
Message-ID: <CACs7g=D3c+N6vugtOwmVvA3G+0uFiyYd9bq5hUX-xwkPnPcWbA@mail.gmail.com>

On Thu, Nov 10, 2016 at 12:43 PM, srinivas devaki
<mr.eightnoteight at gmail.com> wrote:
> complexity wise it's O(N), but space complexity is O(N**2) to execute
> this function,

I'm sorry, that is a mistake.
I just skimmed through the itertoolsmodule.c, and it seems like the
space complexity is just O(N), as when tee objects are copied only the
current index is tracked, so the space complexity is just O(N) and
time complexity is also O(N) as tee object copy operation takes O(1)


Regards
Srinivas Devaki
Senior (4th year) student at Indian Institute of Technology (ISM), Dhanbad
Computer Science and Engineering Department
phone: +91 9491 383 249
telegram: @eightnoteight


From steve+comp.lang.python at pearwood.info  Thu Nov 10 03:01:15 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Thu, 10 Nov 2016 19:01:15 +1100
Subject: N-grams
References: <582326bf$0$1600$c3e8da3$5496439d@news.astraweb.com>
 <87r36kf5rf.fsf@nightsong.com> <694d8ddb-c676-21eb-343b-511851a9453f@web.de>
 <mailman.66.1478760927.30068.python-list@python.org>
Message-ID: <5824294c$0$1515$c3e8da3$5496439d@news.astraweb.com>

On Thursday 10 November 2016 17:53, Wolfram Hinderer wrote:

[...]
> 1. The startup looks slightly ugly to me.
> 2. If n is large, tee has to maintain a lot of unnecessary state.

But n should never be large.

If practice, n-grams are rarely larger than n=3. Occasionally you might use n=4 
or even n=5, but I can't imagine using n=20 in practice, let alone the example 
you show of n=500.

See, for example: 

http://stackoverflow.com/a/10382221

In practice, large n n-grams run into three problems:

- for word-based n-grams, n=3 is about the maximum needed;

- for other applications, n can be moderately large, but as n-grams are a
  kind of auto-correlation function, and few data sets are auto-correlated
  *that* deeply, you still rarely need large values of n;

- there is the problem of sparse data and generating a good training corpus.

For n=10, and just using ASCII letters (lowercase only), there are 26**10 = 
141167095653376 possible 10-grams. Where are you going to find a text that 
includes more than a tiny fraction of those?



-- 
Steven
299792.458 km/s ? not just a good idea, it?s the law!



From __peter__ at web.de  Thu Nov 10 03:56:40 2016
From: __peter__ at web.de (Peter Otten)
Date: Thu, 10 Nov 2016 09:56:40 +0100
Subject: N-grams
References: <582326bf$0$1600$c3e8da3$5496439d@news.astraweb.com>
 <CACs7g=C=GTyEwgWnwbRA7XpOQHeT=VvUcntGGhQSBoDa5x5VQw@mail.gmail.com>
Message-ID: <o01co6$e9e$1@blaine.gmane.org>

srinivas devaki wrote:

Interesting approach.

> def myngrams(iterable, n=2):
>     t = list(tee(iterable, 1))

I don't think I've seen tee(iterable, 1) before. Did you do this for 
aesthetic reasons or is there an advantage over

      t = [iter(iterable)]

?

>     for _ in range(n - 1):
>         t.extend(tee(t.pop()))
>         next(t[-1], None)
>     return zip(*t)
 




From __peter__ at web.de  Thu Nov 10 03:59:03 2016
From: __peter__ at web.de (Peter Otten)
Date: Thu, 10 Nov 2016 09:59:03 +0100
Subject: N-grams
References: <582326bf$0$1600$c3e8da3$5496439d@news.astraweb.com>
 <87r36kf5rf.fsf@nightsong.com>
Message-ID: <o01csj$e9e$2@blaine.gmane.org>

Paul Rubin wrote:

> This can probably be cleaned up some:
> 
>     from itertools import islice
>     from collections import deque
> 
>     def ngram(n, seq):
>         it = iter(seq)
>         d = deque(islice(it, n))
>         if len(d) != n:
>             return
>         for s in it:
>             yield tuple(d)
>             d.popleft()
>             d.append(s)
>         if len(d) == n:
>             yield tuple(d)
> 
>     def test():
>         xs = range(20)
>         for a in ngram(5, xs):
>             print a
> 
>     test()

I started with

def ngrams2(items, n):
    items = iter(items)
    d = deque(islice(items, n-1), maxlen=n)
    for item in items:
        d.append(item)
        yield tuple(d)

and then tried a few dirty tricks, but nothing except omitting tuple(d) 
brought performance near Steven's version.

Just for fun, here's the obligatory oneliner:

def ngrams1(items, n):
    return zip(*(islice(it, i, None) for i, it in enumerate(tee(items, n))))

Be aware that the islice() overhead is significant (I wonder if the islice() 
implementation could be tweaked to reduce that).



From wolfgang.maier at biologie.uni-freiburg.de  Thu Nov 10 04:00:08 2016
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Thu, 10 Nov 2016 10:00:08 +0100
Subject: is modulefinder.ModuleFinder working at all?
In-Reply-To: <5823b909$0$1611$c3e8da3$5496439d@news.astraweb.com>
References: <o0038k$fcg$1@blaine.gmane.org>
 <mailman.48.1478725793.30068.python-list@python.org>
 <5823b909$0$1611$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o01cuk$dcn$1@blaine.gmane.org>

On 10.11.2016 01:02, Steve D'Aprano wrote:
> On Thu, 10 Nov 2016 08:08 am, Wolfgang Maier wrote:
>
>> Hi,
>>
>> I just used the stdlib's modulefinder.ModuleFinder (intended to find
>> modules used by a script) for the first time in my life and it just
>> doesn't seem to work like documented at all.
>> Not sure what is going on, but if I try the usage example from
>> https://docs.python.org/3/library/modulefinder.html
>> it's reporting every single module from the stdlib whether imported or
>> not!
>
> I see what you mean, but it's not quite *every* module. After I run the
> example from the docs, I get 197 modules, and here's at least two that
> aren't included:
>
> py> len(finder.modules)
> 197
> py> 'statistics' in finder.modules
> False
> py> 'cmath' in finder.modules
> False
>
>
> Curiously, it includes modules that aren't cached in sys.modules:
>
> py> len(finder.modules.keys() - sys.modules.keys())
> 107
>
>
> So I'm not sure how that's possible.
>
> According to the example module, it imports re and itertools. Both of those
> have already been imported, and so will be cached in sys.modules, as will
> all their dependencies. So I don't see how it is possible that ModuleFinder
> can find dependencies that aren't cached.
>
> Theoretically, of course some dependency might import a bunch of modules,
> then delete them from sys.modules. If it were only one or two, I'd believe
> that. But not 107 of them.
>
> After running the module finder on the sample file, I ran the report()
> method to get a nicer display of the imported modules:
>
> py> finder = ModuleFinder()
> py> finder.run_script('/tmp/bacon.py')
> py> finder.report()
>
>   Name                      File
>   ----                      ----
> m __future__                /usr/local/lib/python3.5/__future__.py
> m __main__                  /tmp/bacon.py
> m _ast
> m _bootlocale               /usr/local/lib/python3.5/_bootlocale.py
> m _bz2                      /usr/local/lib/python3.5/lib-dynload/_bz2.cpython-35m-i386-linux-gnu.so
> [...]
>
>
> which shows the unittest package being loaded, which is pretty dubious.
>
>
> On the other hand, here's a simpler example which seems to work fine:
>
>
> py> with open('/tmp/do_little.py', 'w') as f:
> ...     f.write('import math\n')
> ...
> 12
> py> finder = ModuleFinder()
> py> finder.run_script('/tmp/do_little.py')
> py> finder.report()
>
>   Name                      File
>   ----                      ----
> m __main__                  /tmp/do_little.py
> m math                      /usr/local/lib/python3.5/lib-dynload/math.cpython-35m-i386-linux-gnu.so
>
>
> So I'm not really sure what's going on.
>

Thanks for the detailed analysis.
You are right that it seems to work with certain imports. I verified 
your example with import math and found a few additional examples.

Things seem to work with imports of any of the following (alone or in 
combination):
io, itertools, math, sys

OTOH, I doubt that it reports correctly with:
os, heapq, operator, re

Personally, I fail to see a pattern here, but maybe somebody else can 
explain that behaviour.
I'd really like to report this to the bug tracker, but at the moment I'm 
not sure as what exactly. It seems that at least the documentation is 
incorrect since the usage example there is not working.

Best,
Wolfgang



From alister.ware at ntlworld.com  Thu Nov 10 05:13:47 2016
From: alister.ware at ntlworld.com (alister)
Date: Thu, 10 Nov 2016 10:13:47 GMT
Subject: Python rules!
References: <b6a23c22-748c-4da1-8c1c-512db3509f53@googlegroups.com>
 <8c2e8884-07a9-44fb-9031-083989642bbe@googlegroups.com>
 <f2f99305-a32e-87a3-62ea-6de44ac177ac@gmail.com>
 <CALwzidm_yzidDM6WhOW2h2PVSuz3yzYbgCfYFLDoV=yvUCXxBg@mail.gmail.com>
 <CANc-5Uxoi-FifO=X5EGkwLg1Gaz-wZ8aO_oQi46bb8Hfuyd3PQ@mail.gmail.com>
 <CANc-5Uxx7DYOZR7jTWQ3ap58RweQ=XXGX5gfh-7QmZFxtpnvXA@mail.gmail.com>
 <CANc-5Uw4Dpm5qOn+g7dY6jz5WYWC7y--aDJ15rOqLpN8trd4Tg@mail.gmail.com>
 <CAPTjJmokiYKEGBNJOQkc1k8qBcKmAJmSJ_bTeOsFYooZYz0ufA@mail.gmail.com>
 <CANc-5Uw39CMC9as35=9A34DYtL11h-sW51O0msNMLZ8=OftQhw@mail.gmail.com>
 <mailman.63.1478745955.30068.python-list@python.org>
Message-ID: <vLXUz.35568$wO.11297@fx37.am4>

On Wed, 09 Nov 2016 20:45:45 -0600, Skip Montanaro wrote:

> On Wed, Nov 9, 2016 at 7:53 PM, Chris Angelico <rosuav at gmail.com> wrote:
>> It's called Jython. :)
> 
> Well, sure, but that didn't look enough like Python, so no chance that I
> would mistake it for Jython. I suspect that whoever worked out that
> arrangement of semicolons and braces had some help from her tools.
> 
> Skip

i think whoever did that WAS a tool



-- 
How long does it take a DEC field service engineer to change a lightbulb?

It depends on how many bad ones he brought with him.


From bc at freeuk.com  Thu Nov 10 08:43:27 2016
From: bc at freeuk.com (BartC)
Date: Thu, 10 Nov 2016 13:43:27 +0000
Subject: Help me cythonize a python routine!
In-Reply-To: <f6846ad5-ff29-4b4f-b7ec-f0d2ef1a6840@googlegroups.com>
References: <nvjm44$8rn$2@dont-email.me> <nvvtmr$2hp$1@dont-email.me>
 <f6846ad5-ff29-4b4f-b7ec-f0d2ef1a6840@googlegroups.com>
Message-ID: <o01th3$94m$1@dont-email.me>

On 09/11/2016 21:25, breamoreboy at gmail.com wrote:
> On Wednesday, November 9, 2016 at 7:34:41 PM UTC, BartC wrote:

> However according to your mindset nothing matters provided it's fast,
 > accuracy does not matter to users.

> Hence your recent comment on another thread about converting invalid integer entries into zero.

That's by design. And it's not unprecedented; using C's 'atoi()' 
function (convert a string to an integer), then inputs of "" and "ABC" 
both return 0. Input of "123ABC" returns 123.

And in the context that was being discussed, it was desired that hitting 
end-of-line while attempting to read more items should return null 
values such as 0, 0.0 or "".

For any more refined behaviour, values can be read differently (read as 
strings then converted with routines that do more validation).

Effectively, the conversion looks at the entire line buffer contents 
remaining, and converts the first integer encountered, if any, then 
removes that and any terminator from the buffer.

If I try something like that in Python, trying to convert an integer at 
the beginning of a string, it fails: int("123,456"), int("123 456"),
int("63924 the"). It will need processing first so that the parameter 
comprises only the thing we're trying to read.

Anyway the comparison between Python and that particular /lower-level/ 
language is moot as Python apparently doesn't have a way of directly 
reading items from an input line, either from a console or from a file.

You have to read a line as one string then apply DIY conversions. Which 
the other language could equally do.

So it's a detail. The other language could also have chosen to use 
exceptions to signal such events (no integer following, end of input, 
invalid terminator etc). I chose to keep it simple and to make it work 
the same way it's always done.

-- 
Bartc


From ethan at stoneleaf.us  Thu Nov 10 10:16:17 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 10 Nov 2016 07:16:17 -0800
Subject: Why I took October off from OSS volunteering
In-Reply-To: <68592351-721b-43e9-a8a1-815e850097dd@googlegroups.com>
References: <68592351-721b-43e9-a8a1-815e850097dd@googlegroups.com>
Message-ID: <58248F41.1090006@stoneleaf.us>

On 11/09/2016 11:35 PM, breamoreboy at gmail.com wrote:

> http://www.snarky.ca/why-i-took-october-off-from-oss-volunteering

Good article, Mark, thanks for sharing.

--
~Ethan~


From torriem at gmail.com  Thu Nov 10 10:36:30 2016
From: torriem at gmail.com (Michael Torrie)
Date: Thu, 10 Nov 2016 08:36:30 -0700
Subject: I want to insert beacon scan result in to a database using python
 and mysql
In-Reply-To: <nvr82c10l2vtsh0cftspi31ordoft18fag@4ax.com>
References: <8336e495-e6e1-4aed-bd82-91a764918768@googlegroups.com>
 <nvr82c10l2vtsh0cftspi31ordoft18fag@4ax.com>
Message-ID: <2a45a828-f35b-299e-888c-7bd5942225d7@gmail.com>

On 11/10/2016 06:15 AM, Dennis Lee Bieber wrote:
> On Wed, 9 Nov 2016 21:05:50 -0800 (PST), sudeeratechneed at gmail.com
> declaimed the following:
> 
>>
>> sql = "insert into beacon VALUES(null, '%s')" % \
>> (beacon)
>>
> 	DON'T DO THAT...

Wouldn't hurt to include a brief why on this, and the right way to do
this.  The why is, of course, that this operation is vulnerable to SQL
injection. This should be avoided as a matter of practice, even if
you're not taking input from anyone but yourself.  The correct way to do
this is to use a prepared statement.  And of course the relevant xkcd
is:  https://xkcd.com/327/




From mr.eightnoteight at gmail.com  Thu Nov 10 10:52:40 2016
From: mr.eightnoteight at gmail.com (srinivas devaki)
Date: Thu, 10 Nov 2016 21:22:40 +0530
Subject: N-grams
In-Reply-To: <o01co6$e9e$1@blaine.gmane.org>
References: <582326bf$0$1600$c3e8da3$5496439d@news.astraweb.com>
 <CACs7g=C=GTyEwgWnwbRA7XpOQHeT=VvUcntGGhQSBoDa5x5VQw@mail.gmail.com>
 <o01co6$e9e$1@blaine.gmane.org>
Message-ID: <CACs7g=CZ63HnBQdTuhioFdU+LEq7OSWNgHSwyzphn0FQdp-3qA@mail.gmail.com>

On Thu, Nov 10, 2016 at 2:26 PM, Peter Otten <__peter__ at web.de> wrote:
>
> I don't think I've seen tee(iterable, 1) before. Did you do this for
> aesthetic reasons or is there an advantage over
>
>       t = [iter(iterable)]

Yeah just to be aesthetic, there's no extra advantage over that as
with n=1 tee just returns a wrapper around the iterable.


Regards
Srinivas Devaki
Senior (4th year) student at Indian Institute of Technology (ISM), Dhanbad
Computer Science and Engineering Department
phone: +91 9491 383 249
telegram: @eightnoteight


From skip.montanaro at gmail.com  Thu Nov 10 11:17:39 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Thu, 10 Nov 2016 10:17:39 -0600
Subject: Python rules!
In-Reply-To: <vLXUz.35568$wO.11297@fx37.am4>
References: <b6a23c22-748c-4da1-8c1c-512db3509f53@googlegroups.com>
 <8c2e8884-07a9-44fb-9031-083989642bbe@googlegroups.com>
 <f2f99305-a32e-87a3-62ea-6de44ac177ac@gmail.com>
 <CALwzidm_yzidDM6WhOW2h2PVSuz3yzYbgCfYFLDoV=yvUCXxBg@mail.gmail.com>
 <CANc-5Uxoi-FifO=X5EGkwLg1Gaz-wZ8aO_oQi46bb8Hfuyd3PQ@mail.gmail.com>
 <CANc-5Uxx7DYOZR7jTWQ3ap58RweQ=XXGX5gfh-7QmZFxtpnvXA@mail.gmail.com>
 <CANc-5Uw4Dpm5qOn+g7dY6jz5WYWC7y--aDJ15rOqLpN8trd4Tg@mail.gmail.com>
 <CAPTjJmokiYKEGBNJOQkc1k8qBcKmAJmSJ_bTeOsFYooZYz0ufA@mail.gmail.com>
 <CANc-5Uw39CMC9as35=9A34DYtL11h-sW51O0msNMLZ8=OftQhw@mail.gmail.com>
 <mailman.63.1478745955.30068.python-list@python.org>
 <vLXUz.35568$wO.11297@fx37.am4>
Message-ID: <CANc-5Ux_v-vK7eAeecZzj2BAe5PYZrG=ygrBT_=2Vja3dL=rvQ@mail.gmail.com>

Alister> i think whoever did that WAS a tool

Perhaps, but maybe she is a Python programmer forced to write Java
(not Jython). If so, props to her for making the best of a bad
situation. :-)

Skip


From rosuav at gmail.com  Thu Nov 10 13:32:10 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 11 Nov 2016 05:32:10 +1100
Subject: I want to insert beacon scan result in to a database using python
 and mysql
In-Reply-To: <2a45a828-f35b-299e-888c-7bd5942225d7@gmail.com>
References: <8336e495-e6e1-4aed-bd82-91a764918768@googlegroups.com>
 <nvr82c10l2vtsh0cftspi31ordoft18fag@4ax.com>
 <2a45a828-f35b-299e-888c-7bd5942225d7@gmail.com>
Message-ID: <CAPTjJmpFVjHE3ZucXVEz0POgjJ8JUDiaZTu1Y_mAO-f5s4O=1g@mail.gmail.com>

On Fri, Nov 11, 2016 at 2:36 AM, Michael Torrie <torriem at gmail.com> wrote:
> On 11/10/2016 06:15 AM, Dennis Lee Bieber wrote:
>> On Wed, 9 Nov 2016 21:05:50 -0800 (PST), sudeeratechneed at gmail.com
>> declaimed the following:
>>
>>>
>>> sql = "insert into beacon VALUES(null, '%s')" % \
>>> (beacon)
>>>
>>       DON'T DO THAT...
>
> Wouldn't hurt to include a brief why on this, and the right way to do
> this.  The why is, of course, that this operation is vulnerable to SQL
> injection. This should be avoided as a matter of practice, even if
> you're not taking input from anyone but yourself.  The correct way to do
> this is to use a prepared statement.  And of course the relevant xkcd
> is:  https://xkcd.com/327/

The easiest way is to use a parameterized query:

cur.execute("insert into beacon VALUES(null, %s)", (beacon,))

I don't understand why so many people conflate parameterized with
prepared. "Prepared statements" have a two-step execution.
"Parameterized queries" needn't.

ChrisA


From torriem at gmail.com  Thu Nov 10 15:15:39 2016
From: torriem at gmail.com (Michael Torrie)
Date: Thu, 10 Nov 2016 13:15:39 -0700
Subject: I want to insert beacon scan result in to a database using python
 and mysql
In-Reply-To: <CAPTjJmpFVjHE3ZucXVEz0POgjJ8JUDiaZTu1Y_mAO-f5s4O=1g@mail.gmail.com>
References: <8336e495-e6e1-4aed-bd82-91a764918768@googlegroups.com>
 <nvr82c10l2vtsh0cftspi31ordoft18fag@4ax.com>
 <2a45a828-f35b-299e-888c-7bd5942225d7@gmail.com>
 <CAPTjJmpFVjHE3ZucXVEz0POgjJ8JUDiaZTu1Y_mAO-f5s4O=1g@mail.gmail.com>
Message-ID: <cf568933-2684-54be-a643-5bbe55f868bf@gmail.com>

On 11/10/2016 11:32 AM, Chris Angelico wrote:
> The easiest way is to use a parameterized query:
> 
> cur.execute("insert into beacon VALUES(null, %s)", (beacon,))
> 
> I don't understand why so many people conflate parameterized with
> prepared. "Prepared statements" have a two-step execution.
> "Parameterized queries" needn't.

Good point. Yes that's what I was going for but forgot the term.



From tumblytumbler at gmail.com  Thu Nov 10 16:37:59 2016
From: tumblytumbler at gmail.com (Keenan C)
Date: Thu, 10 Nov 2016 13:37:59 -0800
Subject: Fwd: Error message
In-Reply-To: <CAP2JEN50vkLBQPwjpnMu-2Q-mqon=jv340OEWDMKYTFqrM-Vuw@mail.gmail.com>
References: <CAP2JEN50vkLBQPwjpnMu-2Q-mqon=jv340OEWDMKYTFqrM-Vuw@mail.gmail.com>
Message-ID: <CAP2JEN4QuYNCXAmTwRU-XOZJ6KAKuB3Jt5cGhAe8eth3pkyXMw@mail.gmail.com>

To whom this may concern,

I am continuously receiving this error after the installation of Python
3.5.2.  The purpose of using this program is for a class I am currently
enrolled in at a University.  (I am running Windows 7 Home Premium 64-bit
paired with an i3-2100 processor and 6 gb of ram.  I don't believe the
specifications are a problems but I have listed them just in case.)
 Although this computer is not new, it has recently been factory reset.  I
have tried multiple times installing and uninstalling this program and
restarting the computer.  If you would be able to help me figure this out,
I would very much appreciate it.

Thank you,
Keenan Chu
[image: Inline image 1]


From thorsten at thorstenkampe.de  Thu Nov 10 16:58:07 2016
From: thorsten at thorstenkampe.de (Thorsten Kampe)
Date: Thu, 10 Nov 2016 22:58:07 +0100
Subject: Windows: subprocess won't run different Python interpreter
Message-ID: <o02qhj$uh5$1@blaine.gmane.org>

Hi,

I'm trying to run a script with a different Python version by 
extending the path variable and executing "python.exe". It looks like 
subprocess will always run the current executing Python.

The following snippet demonstrates the problem:
"""
import os, subprocess
os.environ['PATH'] = ''
print(subprocess.check_output(['python.exe', '-V']))
"""

It outputs the version of the current executing Python interpreter 
although it should generate an error because Python is not in the 
PATH.

Thorsten



From tomuxiong at gmx.com  Thu Nov 10 17:07:35 2016
From: tomuxiong at gmx.com (Thomas Nyberg)
Date: Thu, 10 Nov 2016 17:07:35 -0500
Subject: Windows: subprocess won't run different Python interpreter
In-Reply-To: <o02qhj$uh5$1@blaine.gmane.org>
References: <o02qhj$uh5$1@blaine.gmane.org>
Message-ID: <c46708cd-343e-fb5d-15d4-0c028d87d1f7@gmx.com>

On 11/10/2016 04:58 PM, Thorsten Kampe wrote:
> Hi,
>
> I'm trying to run a script with a different Python version by
> extending the path variable and executing "python.exe". It looks like
> subprocess will always run the current executing Python.
>
 > [...]
 >
> Thorsten
>

Have you tried using the full path to the other binary?

Cheers,
Thomas


From thorsten at thorstenkampe.de  Thu Nov 10 17:32:41 2016
From: thorsten at thorstenkampe.de (Thorsten Kampe)
Date: Thu, 10 Nov 2016 23:32:41 +0100
Subject: Windows: subprocess won't run different Python interpreter
References: <o02qhj$uh5$1@blaine.gmane.org>
 <c46708cd-343e-fb5d-15d4-0c028d87d1f7@gmx.com>
Message-ID: <o02sid$ufn$1@blaine.gmane.org>

* Thomas Nyberg (Thu, 10 Nov 2016 17:07:35 -0500)
> 
> On 11/10/2016 04:58 PM, Thorsten Kampe wrote:
> > Hi,
> >
> > I'm trying to run a script with a different Python version by
> > extending the path variable and executing "python.exe". It looks like
> > subprocess will always run the current executing Python.
> >
>  > [...]
>  >
> > Thorsten
> >
> 
> Have you tried using the full path to the other binary?

Yes. That works. But it's not like subprocess should work.

Thorsten



From tomuxiong at gmx.com  Thu Nov 10 17:46:06 2016
From: tomuxiong at gmx.com (Thomas Nyberg)
Date: Thu, 10 Nov 2016 17:46:06 -0500
Subject: Windows: subprocess won't run different Python interpreter
In-Reply-To: <o02sid$ufn$1@blaine.gmane.org>
References: <o02qhj$uh5$1@blaine.gmane.org>
 <c46708cd-343e-fb5d-15d4-0c028d87d1f7@gmx.com>
 <o02sid$ufn$1@blaine.gmane.org>
Message-ID: <803c59f4-3f55-1c77-78a9-a7b702fe3e62@gmx.com>

On 11/10/2016 05:32 PM, Thorsten Kampe wrote:
> Yes. That works. But it's not like subprocess should work.
>

It certainly is odd. I can at least confirm that when I try to run your 
code I get the error that you're expecting, but I run debian.

Have you tried using os.unsetenv()?

https://docs.python.org/2/library/os.html#os.unsetenv

Cheers,
Thomas


From eryksun at gmail.com  Thu Nov 10 18:04:02 2016
From: eryksun at gmail.com (eryk sun)
Date: Thu, 10 Nov 2016 23:04:02 +0000
Subject: Windows: subprocess won't run different Python interpreter
In-Reply-To: <o02qhj$uh5$1@blaine.gmane.org>
References: <o02qhj$uh5$1@blaine.gmane.org>
Message-ID: <CACL+1avW6AtymQ85J_rMgwWon295N_zbhpwV4WDi=ruQ_Stffg@mail.gmail.com>

On Thu, Nov 10, 2016 at 9:58 PM, Thorsten Kampe
<thorsten at thorstenkampe.de> wrote:
>
> I'm trying to run a script with a different Python version by
> extending the path variable and executing "python.exe". It looks like
> subprocess will always run the current executing Python.

WinAPI CreateProcess checks the application directory, current
directory (an insecure legacy default), %SystemRoot%\System32,
%SystemRoot%\System, %SystemRoot%, and then the directories in %PATH%.
This is listed in the documentation of the lpCommandLine parameter on
MSDN [1].

Let's take a look with a breakpoint set on WinAPI SearchPath [2]:

    >>> subprocess.Popen('python -c 42').wait()
    Breakpoint 0 hit
    KERNELBASE!SearchPathW:
    00007fff`4b344f60 488bc4          mov     rax,rsp

In the x64 ISA, the lpPath parameter is passed in register rcx:

    0:000> du @rcx L4D
    00000241`9a74dbe0  "C:\Program Files\Python35;.;C:\W"
    00000241`9a74dc20  "indows\SYSTEM32;C:\Windows\syste"
    00000241`9a74dc60  "m;C:\Windows;"
    0:000> g
    0

Note that the second entry is ".", which is the current directory.
Setting %NoDefaultCurrentDirectoryInExePath% [3] removes the current
directory from the search path:

    >>> os.environ['NoDefaultCurrentDirectoryInExePath'] = '1'
    >>> subprocess.Popen('python -c 42').wait()
    Breakpoint 0 hit
    KERNELBASE!SearchPathW:
    00007fff`4b344f60 488bc4          mov     rax,rsp

    0:000> du @rcx L4B
    00000241`99e43f90  "C:\Program Files\Python35;C:\Win"
    00000241`99e43fd0  "dows\SYSTEM32;C:\Windows\system;"
    00000241`99e44010  "C:\Windows;"

[1]: https://msdn.microsoft.com/en-us/library/ms682425
[2]: https://msdn.microsoft.com/en-us/library/aa365527
[3]: https://msdn.microsoft.com/en-us/library/ms684269


From eryksun at gmail.com  Thu Nov 10 18:34:12 2016
From: eryksun at gmail.com (eryk sun)
Date: Thu, 10 Nov 2016 23:34:12 +0000
Subject: Error message
In-Reply-To: <CAP2JEN4QuYNCXAmTwRU-XOZJ6KAKuB3Jt5cGhAe8eth3pkyXMw@mail.gmail.com>
References: <CAP2JEN50vkLBQPwjpnMu-2Q-mqon=jv340OEWDMKYTFqrM-Vuw@mail.gmail.com>
 <CAP2JEN4QuYNCXAmTwRU-XOZJ6KAKuB3Jt5cGhAe8eth3pkyXMw@mail.gmail.com>
Message-ID: <CACL+1ati+6unmL09_3eVHOHQ89jXv-M-AzcKYrJuDTQund1QjQ@mail.gmail.com>

On Thu, Nov 10, 2016 at 9:37 PM, Keenan C <tumblytumbler at gmail.com> wrote:
>
> I am continuously receiving this error after the installation of Python
> 3.5.2.  The purpose of using this program is for a class I am currently
> enrolled in at a University.  (I am running Windows 7 Home Premium 64-bit
> paired with an i3-2100 processor and 6 gb of ram.  I don't believe the
> specifications are a problems but I have listed them just in case.)
>  Although this computer is not new, it has recently been factory reset.  I
> have tried multiple times installing and uninstalling this program and
> restarting the computer.
>
> [image: Inline image 1]

python-list is text-only and drops message attachments. Please copy
and paste or transcribe the error message text and error code.


From python at mrabarnett.plus.com  Thu Nov 10 18:40:46 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Thu, 10 Nov 2016 23:40:46 +0000
Subject: Fwd: Error message
In-Reply-To: <CAP2JEN4QuYNCXAmTwRU-XOZJ6KAKuB3Jt5cGhAe8eth3pkyXMw@mail.gmail.com>
References: <CAP2JEN50vkLBQPwjpnMu-2Q-mqon=jv340OEWDMKYTFqrM-Vuw@mail.gmail.com>
 <CAP2JEN4QuYNCXAmTwRU-XOZJ6KAKuB3Jt5cGhAe8eth3pkyXMw@mail.gmail.com>
Message-ID: <97bbbe16-5ccd-d4b9-d175-b36f54b212f9@mrabarnett.plus.com>

On 2016-11-10 21:37, Keenan C wrote:
> To whom this may concern,
>
> I am continuously receiving this error after the installation of Python
> 3.5.2.  The purpose of using this program is for a class I am currently
> enrolled in at a University.  (I am running Windows 7 Home Premium 64-bit
> paired with an i3-2100 processor and 6 gb of ram.  I don't believe the
> specifications are a problems but I have listed them just in case.)
>  Although this computer is not new, it has recently been factory reset.  I
> have tried multiple times installing and uninstalling this program and
> restarting the computer.  If you would be able to help me figure this out,
> I would very much appreciate it.
>
> Thank you,
> Keenan Chu
> [image: Inline image 1]
>
This list doesn't support attachments, so I'm guessing that it's 
complaining that it can't find "api-ms-win-crd-runtime-l1-1-0.dll".

You should already have it if you've kept your PC up to date via Windows 
Update, but if you haven't, read this:

https://support.microsoft.com/en-us/kb/3118401



From torriem at gmail.com  Thu Nov 10 20:29:53 2016
From: torriem at gmail.com (Michael Torrie)
Date: Thu, 10 Nov 2016 18:29:53 -0700
Subject: I want to insert beacon scan result in to a database using python
 and mysql
In-Reply-To: <2f6a2ch191p5deec3ojaq7p3grolqujjnn@4ax.com>
References: <8336e495-e6e1-4aed-bd82-91a764918768@googlegroups.com>
 <nvr82c10l2vtsh0cftspi31ordoft18fag@4ax.com>
 <2a45a828-f35b-299e-888c-7bd5942225d7@gmail.com>
 <CAPTjJmpFVjHE3ZucXVEz0POgjJ8JUDiaZTu1Y_mAO-f5s4O=1g@mail.gmail.com>
 <cf568933-2684-54be-a643-5bbe55f868bf@gmail.com>
 <2f6a2ch191p5deec3ojaq7p3grolqujjnn@4ax.com>
Message-ID: <49a32a3c-3fcc-21ea-d948-1d98b5add605@gmail.com>

On 11/10/2016 06:10 PM, Dennis Lee Bieber wrote:
> {I could swear I'd included an example of a parameterized query in my
> response... I didn't want to go into the details of "SQL injection attack"
> as, based on the rest of the OPs post, it would have needed a large
> explanation... And the biggest flaw was improper indentation for the
> database access}

You did indeed.  My bad.


From jobmattcon at gmail.com  Thu Nov 10 20:34:34 2016
From: jobmattcon at gmail.com (meInvent bbird)
Date: Thu, 10 Nov 2016 17:34:34 -0800 (PST)
Subject: what is the procedure or how to plan how many nodes of dispy need for
 dsolve differential system in amazon cloud in limited time such as 1
 hour, 2 hours.etc
Message-ID: <dc5c6181-b036-4fca-9caa-2c1b85675494@googlegroups.com>

what is the procedure or how to plan how many nodes of dispy need for dsolve differential system in amazon cloud in limited time such as 1 hour, 2 hours.etc ?


#For Amazon Linux, the user name is ec2-user. For RHEL5, the user name is either root or ec2-user. 
#For Ubuntu, the user name is ubuntu. For Fedora, the user name is either fedora or ec2-user. 
#For SUSE Linux, the user name is either root or ec2-user. 
#Otherwise, if ec2-user and root don't work, check with your AMI provider.

import random, dispy
import ast 
from __future__ import division 
from sympy import * 
x, y, z, t = symbols('x y z t') 
k, m, n = symbols('k m n', integer=True) 
f, g, h = symbols('f g h', cls=Function) 
import inspect 

def compute(n): # executed on nodes
    import random, time, socket
    name = socket.gethostname()
    cur_best = 1
    for ii in range(n[0],n[0]):
        for jj in range(n[1],n[1]):
            for kk in range(n[2],n[3],100):  
                #assume dsolve with sympy for differential system   
                dispy_provisional_result((name, r))
                cur_best = r
                time.sleep(0.1)
                # final result
    return (name, cur_best)

def job_callback(job): # executed at the client
    if job.status == dispy.DispyJob.ProvisionalResult:
        #if job.result[1] < 0.005:
            # acceptable result; terminate jobs
            print('%s computed: %s %s %s %s' % (job.result[0], job.result[1], job.result[2], job.result[3], job.result[4]))
            # 'jobs' and 'cluster' are created in '__main__' below
            for j in jobs:
                if j.status in [dispy.DispyJob.Created, dispy.DispyJob.Running,
                                dispy.DispyJob.ProvisionalResult]:
                    cluster.cancel(j)

if __name__ == '__main__':
    #cluster = dispy.JobCluster(compute, callback=job_callback)
    cluster = dispy.JobCluster(compute, nodes=['ec2-35-162-137-237.us-west-2.compute.amazonaws.com'], ip_addr='127.0.0.1', port=51347, node_port=51348, callback=job_callback, keyfile=r"C:\Users\hello\datacenterusekey.ppk")
    jobs = []
    prevk = 1
    count = 0
    for ii in range(1,2):
        for jj in range(1,2000):
            for kk in range(1,2000,100):
                if ii < jj and jj < kk:
                    job = cluster.submit([ii,jj,prevk,kk])
                    prevk = kk
                    if job is None:
                        print('creating job %s failed!' % n)
                        continue
                    job.id = count
                    count = count + 1
                    jobs.append(job)
    cluster.wait()
    cluster.print_status()
    cluster.close()


From vivabs at gmail.com  Thu Nov 10 21:11:29 2016
From: vivabs at gmail.com (Vidyalakshmi Rao)
Date: Fri, 11 Nov 2016 13:11:29 +1100
Subject: Reporting a Bug
Message-ID: <CAG7NVB=XCwtmvvrCEKYMJPjbvpXRrFDOiJ7Om85--8CE1YDmJA@mail.gmail.com>

Hi,

System Specification:

2.7.7 |Anaconda custom (64-bit)| (default, Jun 11 2014, 10:40:02) [MSC
v.1500 64 bit (AMD64)]


*Issue: Spyder hangs in instances while iterating over a list containing
rows with nothing in it.*


    for eachinfo in range(len(textlist)): #remove non-ASCII characters from
the textlist
        textlist[eachinfo] = re.sub(r'[^\x00-\x7F]+','', textlist[eachinfo])

This results in a list *textlist*, which contains some rows with nothing in
it. When such a list is iterated over (as below), Spyder hangs indefinitely.

    for item in textlist:
        print item

While debugging through Spyder (Ctrl+F5), it hangs indefinitely. But, if we
just run(F5), executes without a glitch.
Am i doing something wrong or is it a bug? Is it related to Spyder?
I am a novice to Python. Please direct me if this is not a Python issue but
an Anaconda issue.

Thanks,
Vidya.


From thorsten at thorstenkampe.de  Fri Nov 11 01:01:42 2016
From: thorsten at thorstenkampe.de (Thorsten Kampe)
Date: Fri, 11 Nov 2016 07:01:42 +0100
Subject: Windows: subprocess won't run different Python interpreter
References: <o02qhj$uh5$1@blaine.gmane.org>
 <CACL+1avW6AtymQ85J_rMgwWon295N_zbhpwV4WDi=ruQ_Stffg@mail.gmail.com>
Message-ID: <o03msc$f5i$1@blaine.gmane.org>

* eryk sun (Thu, 10 Nov 2016 23:04:02 +0000)
> 
> On Thu, Nov 10, 2016 at 9:58 PM, Thorsten Kampe
> <thorsten at thorstenkampe.de> wrote:
> >
> > I'm trying to run a script with a different Python version by
> > extending the path variable and executing "python.exe". It looks like
> > subprocess will always run the current executing Python.
> 
> WinAPI CreateProcess checks the application directory, current
> directory (an insecure legacy default), %SystemRoot%\System32,
> %SystemRoot%\System, %SystemRoot%, and then the directories in %PATH%.
> This is listed in the documentation of the lpCommandLine parameter on
> MSDN [1].

I'm aware of that. Python is in F:\PortableApps\Python3x and neither 
the current directory nor the PATH points to that.

Thorsten



From thorsten at thorstenkampe.de  Fri Nov 11 01:05:44 2016
From: thorsten at thorstenkampe.de (Thorsten Kampe)
Date: Fri, 11 Nov 2016 07:05:44 +0100
Subject: Windows: subprocess won't run different Python interpreter
References: <o02qhj$uh5$1@blaine.gmane.org>
 <c46708cd-343e-fb5d-15d4-0c028d87d1f7@gmx.com>
 <o02sid$ufn$1@blaine.gmane.org>
 <803c59f4-3f55-1c77-78a9-a7b702fe3e62@gmx.com>
Message-ID: <o03n3t$f5i$2@blaine.gmane.org>

* Thomas Nyberg (Thu, 10 Nov 2016 17:46:06 -0500)
> 
> On 11/10/2016 05:32 PM, Thorsten Kampe wrote:
> > Yes. That works. But it's not like subprocess should work.
> >
> 
> It certainly is odd. I can at least confirm that when I try to run your 
> code I get the error that you're expecting, but I run debian.
> 
> Have you tried using os.unsetenv()?

I think you are kind of misunderstanding my intention. I'm actually 
trying the opposite: running python.exe by setting F:\PortableApps
\Python3x as the first element in PATH. No Python interpreter is in 
my PATH. So unsetting PATH is just a way to demonstrate that this 
should never ever work.

I'm going to open a bug report.

Thorsten



From eryksun at gmail.com  Fri Nov 11 01:23:50 2016
From: eryksun at gmail.com (eryk sun)
Date: Fri, 11 Nov 2016 06:23:50 +0000
Subject: Windows: subprocess won't run different Python interpreter
In-Reply-To: <o03msc$f5i$1@blaine.gmane.org>
References: <o02qhj$uh5$1@blaine.gmane.org>
 <CACL+1avW6AtymQ85J_rMgwWon295N_zbhpwV4WDi=ruQ_Stffg@mail.gmail.com>
 <o03msc$f5i$1@blaine.gmane.org>
Message-ID: <CACL+1avNwVpdsQ_ixgUj=oSujpmU-Gud2c4B8vDTa0_ymmxBHw@mail.gmail.com>

On Fri, Nov 11, 2016 at 6:01 AM, Thorsten Kampe
<thorsten at thorstenkampe.de> wrote:
> * eryk sun (Thu, 10 Nov 2016 23:04:02 +0000)
>>
>> On Thu, Nov 10, 2016 at 9:58 PM, Thorsten Kampe
>> <thorsten at thorstenkampe.de> wrote:
>> >
>> > I'm trying to run a script with a different Python version by
>> > extending the path variable and executing "python.exe". It looks like
>> > subprocess will always run the current executing Python.
>>
>> WinAPI CreateProcess checks the application directory, current
>> directory (an insecure legacy default), %SystemRoot%\System32,
>> %SystemRoot%\System, %SystemRoot%, and then the directories in %PATH%.
>> This is listed in the documentation of the lpCommandLine parameter on
>> MSDN [1].
>
> I'm aware of that. Python is in F:\PortableApps\Python3x and neither
> the current directory nor the PATH points to that.

That's the application directory, which is the first place
CreateProcess looks (via the SearchPath call), as both of my examples
shows. In my case python.exe is located in the standard 3.5 system
installation path, "C:\Program Files\Python35".

I showed how to remove the current directory from the search. But, in
hindsight, I should have clarified that the working directory was not
related to your problem. It's just the only directory that can be
removed from the implicit search path (AFAIK). I thought about
replying with a clarification, but it seemed clear in context that the
source of the behavior was the application directory.

I know of no API, environment variable, or manifest setting to remove
the application directory from the head of the search path used by
CreateProcess. What you can do is call SearchPath explicitly (via
PyWin32 or ctypes) with just %Path%, or do the search in pure Python
via shutil.which.

Note also that LoadLibrary uses a separate DLL search path that also
defaults to preferring the application directory, but this can be
controlled by calling SetDefaultDllDirectories [1], among other
strategies.

[1]: https://msdn.microsoft.com/en-us/library/hh310515


From thorsten at thorstenkampe.de  Fri Nov 11 03:56:43 2016
From: thorsten at thorstenkampe.de (Thorsten Kampe)
Date: Fri, 11 Nov 2016 09:56:43 +0100
Subject: Windows: subprocess won't run different Python interpreter
References: <o02qhj$uh5$1@blaine.gmane.org>
 <CACL+1avW6AtymQ85J_rMgwWon295N_zbhpwV4WDi=ruQ_Stffg@mail.gmail.com>
 <o03msc$f5i$1@blaine.gmane.org>
 <CACL+1avNwVpdsQ_ixgUj=oSujpmU-Gud2c4B8vDTa0_ymmxBHw@mail.gmail.com>
Message-ID: <o04146$fce$1@blaine.gmane.org>

* eryk sun (Fri, 11 Nov 2016 06:23:50 +0000)
> 
> That's the application directory, which is the first place
> CreateProcess looks (via the SearchPath call), as both of my examples
> shows. In my case python.exe is located in the standard 3.5 system
> installation path, "C:\Program Files\Python35".

Okay, it looks like I read your first answer not thorough enough.

So if the application's directory is always searched then the issue 
should be reproducible with any native (non-Cygwin) Windows 
interpreter:

"""
tcc> \PortableApps\TCC_RT\tcc.exe /c run-TEST.btm
unset PATH
tcc.exe /c ver
TCC: C:\Documents\batch\run-TEST.btm [2]  Unbekannter Befehl 
"tcc.exe"
"""

So TCC can't find itself with an empty PATH. That's how Python 
(subprocess) should also work.

Thorsten



From eryksun at gmail.com  Fri Nov 11 04:55:23 2016
From: eryksun at gmail.com (eryk sun)
Date: Fri, 11 Nov 2016 09:55:23 +0000
Subject: Windows: subprocess won't run different Python interpreter
In-Reply-To: <o04146$fce$1@blaine.gmane.org>
References: <o02qhj$uh5$1@blaine.gmane.org>
 <CACL+1avW6AtymQ85J_rMgwWon295N_zbhpwV4WDi=ruQ_Stffg@mail.gmail.com>
 <o03msc$f5i$1@blaine.gmane.org>
 <CACL+1avNwVpdsQ_ixgUj=oSujpmU-Gud2c4B8vDTa0_ymmxBHw@mail.gmail.com>
 <o04146$fce$1@blaine.gmane.org>
Message-ID: <CACL+1auTV+OJ-ebtADAfAACJk31zWGskC7A2GELS6UhOMg9MjA@mail.gmail.com>

On Fri, Nov 11, 2016 at 8:56 AM, Thorsten Kampe
<thorsten at thorstenkampe.de> wrote:
> * eryk sun (Fri, 11 Nov 2016 06:23:50 +0000)
>>
>> That's the application directory, which is the first place
>> CreateProcess looks (via the SearchPath call), as both of my examples
>> shows. In my case python.exe is located in the standard 3.5 system
>> installation path, "C:\Program Files\Python35".
>
> Okay, it looks like I read your first answer not thorough enough.
>
> So if the application's directory is always searched then the issue
> should be reproducible with any native (non-Cygwin) Windows
> interpreter:
>
> """
> tcc> \PortableApps\TCC_RT\tcc.exe /c run-TEST.btm
> unset PATH
> tcc.exe /c ver
> TCC: C:\Documents\batch\run-TEST.btm [2]  Unbekannter Befehl
> "tcc.exe"
> """
>
> So TCC can't find itself with an empty PATH. That's how Python
> (subprocess) should also work.

If it works like cmd.exe, then it does its own search using %Path% and
%PathExt%. For example:

    C:\>cmd /c "set "PATH=" & cmd"
    'cmd' is not recognized as an internal or external command,
    operable program or batch file.

But why should subprocess.Popen implement its own search like the
shell instead of relying on CreateProcess to search for the
executable? You'd have to come up with an argument to convince the
devs to change the behavior in 3.7.


From info at egenix.com  Fri Nov 11 05:30:31 2016
From: info at egenix.com (eGenix Team: M.-A. Lemburg)
Date: Fri, 11 Nov 2016 11:30:31 +0100
Subject: ANN: eGenix pyOpenSSL Distribution 0.13.16
Message-ID: <58259DC7.4090809@egenix.com>

________________________________________________________________________
ANNOUNCING

                   eGenix.com pyOpenSSL Distribution

                            Version 0.13.16

            An easy-to-install and easy-to-use distribution
            of the pyOpenSSL Python interface for OpenSSL -
           available for Windows, Mac OS X and Unix platforms


This announcement is also available on our web-site for online reading:
http://www.egenix.com/company/news/eGenix-pyOpenSSL-Distribution-0.13.16.html

________________________________________________________________________
INTRODUCTION

The eGenix.com pyOpenSSL Distribution includes everything you need to
get started with SSL in Python.

It comes with an easy-to-use installer that includes the most recent
OpenSSL library versions in pre-compiled form, making your application
independent of OS provided OpenSSL libraries:

    http://www.egenix.com/products/python/pyOpenSSL/

pyOpenSSL is an open-source Python add-on that allows writing SSL/TLS-
aware network applications as well as certificate management tools:

    https://launchpad.net/pyopenssl/

OpenSSL is an open-source implementation of the SSL/TLS protocol:

    http://www.openssl.org/

________________________________________________________________________
NEWS

This new release of the eGenix.com pyOpenSSL Distribution includes the
following updates:

New in OpenSSL
--------------

 * Switched the included OpenSSL libraries to 1.0.2j.

    The OpenSSL 1.0.2 branch will receive long term support (LTS), so
    is an ideal basis for development. See
    https://www.openssl.org/news/secadv/20160926.txt for a complete
    list of security fixes in 1.0.2j. The following fixes are relevant
    for pyOpenSSL applications:

    - CVE-2016-6304 A malicious client can send an excessively large
      OCSP Status Request extension leading to a DoS attack.

    - CVE-2016-6306 Some missing message length checks can result in
      OOB reads, which could be used for DoS attacks.

 * Updated the Mozilla CA root bundle to the current version as of
2016-11-10.

Please see the product changelog for the full set of changes.

    http://www.egenix.com/products/python/pyOpenSSL/changelog.html


pyOpenSSL / OpenSSL Binaries Included
-------------------------------------

In addition to providing sources, we make binaries available that
include both pyOpenSSL and the necessary OpenSSL libraries for all
supported platforms: Windows, Linux, Mac OS X and FreeBSD, for x86 and
x64.

To simplify installation, we have uploaded a web installer to PyPI
which will automatically choose the right binary for your platform, so
a simple

    pip install egenix-pyopenssl

will get you the package with OpenSSL libraries installed. Please see
our installation instructions for details:

    http://www.egenix.com/products/python/pyOpenSSL/#Installation

We have also added .egg-file distribution versions of our eGenix.com
pyOpenSSL Distribution for Windows, Linux and Mac OS X to the
available download options. These make setups using e.g. zc.buildout
and other egg-file based installers a lot easier.

________________________________________________________________________
DOWNLOADS

The download archives and instructions for installing the package can
be found at:

    http://www.egenix.com/products/python/pyOpenSSL/

________________________________________________________________________
UPGRADING

Before installing this version of pyOpenSSL, please make sure that
you uninstall any previously installed pyOpenSSL version. Otherwise,
you could end up not using the included OpenSSL libs.

_______________________________________________________________________
SUPPORT

Commercial support for these packages is available from eGenix.com.
Please see

    http://www.egenix.com/services/support/

for details about our support offerings.

________________________________________________________________________
MORE INFORMATION

For more information about the eGenix pyOpenSSL Distribution, licensing
and download instructions, please visit our web-site or write to
sales at egenix.com.

About eGenix (http://www.egenix.com/):

    eGenix is a Python software project, consulting and product
    company delivering expert services and professional quality
    products for companies, Python users and developers. We specialize
    in database driven applications, large scale software designs and
    integration.

Enjoy,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 11 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/



From thorsten at thorstenkampe.de  Fri Nov 11 05:46:59 2016
From: thorsten at thorstenkampe.de (Thorsten Kampe)
Date: Fri, 11 Nov 2016 11:46:59 +0100
Subject: Windows: subprocess won't run different Python interpreter
References: <o02qhj$uh5$1@blaine.gmane.org>
 <CACL+1avW6AtymQ85J_rMgwWon295N_zbhpwV4WDi=ruQ_Stffg@mail.gmail.com>
 <o03msc$f5i$1@blaine.gmane.org>
 <CACL+1avNwVpdsQ_ixgUj=oSujpmU-Gud2c4B8vDTa0_ymmxBHw@mail.gmail.com>
 <o04146$fce$1@blaine.gmane.org>
 <CACL+1auTV+OJ-ebtADAfAACJk31zWGskC7A2GELS6UhOMg9MjA@mail.gmail.com>
Message-ID: <o047it$m9q$1@blaine.gmane.org>

* eryk sun (Fri, 11 Nov 2016 09:55:23 +0000)
> 
> If it works like cmd.exe, then it does its own search using %Path% 
> and %PathExt%. For example:
> 
>     C:\>cmd /c "set "PATH=" & cmd"
>     'cmd' is not recognized as an internal or external command,
>     operable program or batch file.
> 
> But why should subprocess.Popen implement its own search like the
> shell instead of relying on CreateProcess to search for the
> executable? You'd have to come up with an argument to convince the
> devs to change the behavior in 3.7.

My goal is to verify that other shells/interpreters on Windows work 
the same way as Python when running an application or creating a sub-
process. Cmd does not. What's else there? I have Bash here but that's 
a Cygwin executable. And Cygwin Python does not work like Windows 
Python.

Any ideas?

Thorsten



From daiyueweng at gmail.com  Fri Nov 11 06:17:14 2016
From: daiyueweng at gmail.com (Daiyue Weng)
Date: Fri, 11 Nov 2016 11:17:14 +0000
Subject: update certain key-value pairs of a dict from another dict
Message-ID: <CAF_EDkuib3bNhu2ufeOosEpUH=b6RRsnjD6OGWFkyxrrB=_tSg@mail.gmail.com>

Hi, I have two dicts, e.g.

dict1 = {'A': 'a', 'B': 'b', 'C': 'c'}
dict2 = {'A': 'aa', 'B': 'bb', 'C': 'cc'}

I am wondering how to update dict1 using dict2 that

only keys 'A' and 'B' of dict1 are udpated. It will result in

dict1 = {'A': 'aa', 'B': 'bb', 'C': 'c'}

cheers


From eryksun at gmail.com  Fri Nov 11 06:24:14 2016
From: eryksun at gmail.com (eryk sun)
Date: Fri, 11 Nov 2016 11:24:14 +0000
Subject: Windows: subprocess won't run different Python interpreter
In-Reply-To: <o047it$m9q$1@blaine.gmane.org>
References: <o02qhj$uh5$1@blaine.gmane.org>
 <CACL+1avW6AtymQ85J_rMgwWon295N_zbhpwV4WDi=ruQ_Stffg@mail.gmail.com>
 <o03msc$f5i$1@blaine.gmane.org>
 <CACL+1avNwVpdsQ_ixgUj=oSujpmU-Gud2c4B8vDTa0_ymmxBHw@mail.gmail.com>
 <o04146$fce$1@blaine.gmane.org>
 <CACL+1auTV+OJ-ebtADAfAACJk31zWGskC7A2GELS6UhOMg9MjA@mail.gmail.com>
 <o047it$m9q$1@blaine.gmane.org>
Message-ID: <CACL+1atQ_MuLX2EOLB1QcbqnsVCSi=vpiUk5ePGXBz6DsjH+Hw@mail.gmail.com>

On Fri, Nov 11, 2016 at 10:46 AM, Thorsten Kampe
<thorsten at thorstenkampe.de> wrote:
> * eryk sun (Fri, 11 Nov 2016 09:55:23 +0000)
>>
>> If it works like cmd.exe, then it does its own search using %Path%
>> and %PathExt%. For example:
>>
>>     C:\>cmd /c "set "PATH=" & cmd"
>>     'cmd' is not recognized as an internal or external command,
>>     operable program or batch file.
>>
>> But why should subprocess.Popen implement its own search like the
>> shell instead of relying on CreateProcess to search for the
>> executable? You'd have to come up with an argument to convince the
>> devs to change the behavior in 3.7.
>
> My goal is to verify that other shells/interpreters on Windows work
> the same way as Python when running an application or creating a sub-
> process. Cmd does not. What's else there? I have Bash here but that's
> a Cygwin executable. And Cygwin Python does not work like Windows
> Python.
>
> Any ideas?

PowerShell uses its own search:

    C:\>powershell -c "$Env:Path=''; powershell"
    powershell : The term 'powershell' is not recognized as the name of
    a cmdlet, function, script file, or operable program. Check the
    spelling of the name, or if a path was included, verify that the
    path is correct and try again.
    At line:1 char:15
    + $Env:Path=''; powershell
    +               ~~~~~~~~~~
        + CategoryInfo          : ObjectNotFound: (powershell:String) [],
                                  CommandNotFoundException
        + FullyQualifiedErrorId : CommandNotFoundException

These shells implement their own search because they use %PathExt% to
broaden the search and then call ShellExecuteEx to be able to
execute/open files that CreateProcess cannot. The only file extension
that CreateProcess tries to append is .EXE.

Since subprocess.Popen doesn't call ShellExecuteEx, it would make no
sense for it to use a custom search that tries appending all of the
file extensions in %PathExt%. Thus the only case for Popen to
implement its own search is to avoid the implicit directories that
CreateProcess searches ahead of %Path%. But implicitly searching the
application directory and system directories ahead of %Path% is a
feature for Windows applications, and Python is primarily an
application development language on Windows -- not a system
administration language that takes the place of the shell.


From gvanem at yahoo.no  Fri Nov 11 06:30:48 2016
From: gvanem at yahoo.no (Gisle Vanem)
Date: Fri, 11 Nov 2016 12:30:48 +0100
Subject: Windows: subprocess won't run different Python interpreter
In-Reply-To: <o047it$m9q$1@blaine.gmane.org>
References: <o02qhj$uh5$1@blaine.gmane.org>
 <CACL+1avW6AtymQ85J_rMgwWon295N_zbhpwV4WDi=ruQ_Stffg@mail.gmail.com>
 <o03msc$f5i$1@blaine.gmane.org>
 <CACL+1avNwVpdsQ_ixgUj=oSujpmU-Gud2c4B8vDTa0_ymmxBHw@mail.gmail.com>
 <o04146$fce$1@blaine.gmane.org>
 <CACL+1auTV+OJ-ebtADAfAACJk31zWGskC7A2GELS6UhOMg9MjA@mail.gmail.com>
 <o047it$m9q$1@blaine.gmane.org>
Message-ID: <f8006fb0-085a-1bcb-f6e5-b3e4c5c00cdd@yahoo.no>

Thorsten Kampe wrote:

> My goal is to verify that other shells/interpreters on Windows work 
> the same way as Python when running an application or creating a sub-
> process. Cmd does not. What's else there? I have Bash here but that's 
> a Cygwin executable. And Cygwin Python does not work like Windows 
> Python.
> 
> Any ideas?

Is there a Python.exe in the Registry "App Paths". Either of these:
 HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\App Paths
or
 HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\App Paths

-- 
--gv


From eryksun at gmail.com  Fri Nov 11 06:40:38 2016
From: eryksun at gmail.com (eryk sun)
Date: Fri, 11 Nov 2016 11:40:38 +0000
Subject: Windows: subprocess won't run different Python interpreter
In-Reply-To: <f8006fb0-085a-1bcb-f6e5-b3e4c5c00cdd@yahoo.no>
References: <o02qhj$uh5$1@blaine.gmane.org>
 <CACL+1avW6AtymQ85J_rMgwWon295N_zbhpwV4WDi=ruQ_Stffg@mail.gmail.com>
 <o03msc$f5i$1@blaine.gmane.org>
 <CACL+1avNwVpdsQ_ixgUj=oSujpmU-Gud2c4B8vDTa0_ymmxBHw@mail.gmail.com>
 <o04146$fce$1@blaine.gmane.org>
 <CACL+1auTV+OJ-ebtADAfAACJk31zWGskC7A2GELS6UhOMg9MjA@mail.gmail.com>
 <o047it$m9q$1@blaine.gmane.org>
 <f8006fb0-085a-1bcb-f6e5-b3e4c5c00cdd@yahoo.no>
Message-ID: <CACL+1atc8O0tb7VyeC6rWY2W901vLOOqDaaxcACJV8_iiQZv8w@mail.gmail.com>

On Fri, Nov 11, 2016 at 11:30 AM, Gisle Vanem via Python-list
<python-list at python.org> wrote:
> Thorsten Kampe wrote:
>
>> My goal is to verify that other shells/interpreters on Windows work
>> the same way as Python when running an application or creating a sub-
>> process. Cmd does not. What's else there? I have Bash here but that's
>> a Cygwin executable. And Cygwin Python does not work like Windows
>> Python.
>>
>> Any ideas?
>
> Is there a Python.exe in the Registry "App Paths". Either of these:
>  HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\App Paths
> or
>  HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\App Paths

An "App Paths" key would be used by ShellExecuteEx, not CreateProcess,
so it has no bearing on the behavior of subprocess.Popen with
shell=False.


From python.list at tim.thechases.com  Fri Nov 11 06:59:17 2016
From: python.list at tim.thechases.com (Tim Chase)
Date: Fri, 11 Nov 2016 05:59:17 -0600
Subject: update certain key-value pairs of a dict from another dict
In-Reply-To: <CAF_EDkuib3bNhu2ufeOosEpUH=b6RRsnjD6OGWFkyxrrB=_tSg@mail.gmail.com>
References: <CAF_EDkuib3bNhu2ufeOosEpUH=b6RRsnjD6OGWFkyxrrB=_tSg@mail.gmail.com>
Message-ID: <20161111055917.100de01c@bigbox.christie.dr>

On 2016-11-11 11:17, Daiyue Weng wrote:
> dict1 = {'A': 'a', 'B': 'b', 'C': 'c'}
> dict2 = {'A': 'aa', 'B': 'bb', 'C': 'cc'}
> 
> I am wondering how to update dict1 using dict2 that
> 
> only keys 'A' and 'B' of dict1 are udpated. It will result in
> 
> dict1 = {'A': 'aa', 'B': 'bb', 'C': 'c'}

Use dict1's .update() method:

>>> dict1 = {'A': 'a', 'B': 'b', 'C': 'c'}
>>> dict2 = {'A': 'aa', 'B': 'bb', 'C': 'cc'}
>>> desired = {'A', 'B'}
>>> dict1.update({k:v for k,v in dict2.items() if k in desired})
>>> dict1
{'C': 'c', 'B': 'bb', 'A': 'aa'}


or do it manually

  for k in dict2:
    if k in desired:
      dict1[k] = dict2[k]

-tkc






From __peter__ at web.de  Fri Nov 11 07:29:36 2016
From: __peter__ at web.de (Peter Otten)
Date: Fri, 11 Nov 2016 13:29:36 +0100
Subject: update certain key-value pairs of a dict from another dict
References: <CAF_EDkuib3bNhu2ufeOosEpUH=b6RRsnjD6OGWFkyxrrB=_tSg@mail.gmail.com>
 <20161111055917.100de01c@bigbox.christie.dr>
Message-ID: <o04djd$g9o$1@blaine.gmane.org>

Tim Chase wrote:

> On 2016-11-11 11:17, Daiyue Weng wrote:
>> dict1 = {'A': 'a', 'B': 'b', 'C': 'c'}
>> dict2 = {'A': 'aa', 'B': 'bb', 'C': 'cc'}
>> 
>> I am wondering how to update dict1 using dict2 that
>> 
>> only keys 'A' and 'B' of dict1 are udpated. It will result in
>> 
>> dict1 = {'A': 'aa', 'B': 'bb', 'C': 'c'}
> 
> Use dict1's .update() method:
> 
>>>> dict1 = {'A': 'a', 'B': 'b', 'C': 'c'}
>>>> dict2 = {'A': 'aa', 'B': 'bb', 'C': 'cc'}
>>>> desired = {'A', 'B'}
>>>> dict1.update({k:v for k,v in dict2.items() if k in desired})
>>>> dict1
> {'C': 'c', 'B': 'bb', 'A': 'aa'}
> 
> 
> or do it manually
> 
>   for k in dict2:
>     if k in desired:
>       dict1[k] = dict2[k]

If desired is "small" compared to the dicts:

>>> for k in desired & dict1.keys() & dict2.keys():
...     dict1[k] = dict2[k]
... 
>>> dict1
{'A': 'aa', 'C': 'c', 'B': 'bb'}

The same using update(), with a generator expression that avoids the 
intermediate dict:

>>> dict1 = {'A': 'a', 'B': 'b', 'C': 'c'}
>>> dict1.update((k, dict2[k]) for k in desired & dict1.keys() & 
dict2.keys())
>>> dict1
{'A': 'aa', 'C': 'c', 'B': 'bb'}

As written this will add no new keys to dict1. If you want to allow new keys 
use 

desired & dict2.keys()

as the set of keys.



From ethan at stoneleaf.us  Fri Nov 11 10:57:06 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 11 Nov 2016 07:57:06 -0800
Subject: Windows: subprocess won't run different Python interpreter
In-Reply-To: <CACL+1atQ_MuLX2EOLB1QcbqnsVCSi=vpiUk5ePGXBz6DsjH+Hw@mail.gmail.com>
References: <o02qhj$uh5$1@blaine.gmane.org>
 <CACL+1avW6AtymQ85J_rMgwWon295N_zbhpwV4WDi=ruQ_Stffg@mail.gmail.com>
 <o03msc$f5i$1@blaine.gmane.org>
 <CACL+1avNwVpdsQ_ixgUj=oSujpmU-Gud2c4B8vDTa0_ymmxBHw@mail.gmail.com>
 <o04146$fce$1@blaine.gmane.org>
 <CACL+1auTV+OJ-ebtADAfAACJk31zWGskC7A2GELS6UhOMg9MjA@mail.gmail.com>
 <o047it$m9q$1@blaine.gmane.org>
 <CACL+1atQ_MuLX2EOLB1QcbqnsVCSi=vpiUk5ePGXBz6DsjH+Hw@mail.gmail.com>
Message-ID: <5825EA52.2030407@stoneleaf.us>

On 11/11/2016 03:24 AM, eryk sun wrote:
> On Fri, Nov 11, 2016 at 10:46 AM, Thorsten Kampe wrote:

>> My goal is to verify that other shells/interpreters on Windows work
>> the same way as Python when running an application or creating a sub-
>> process. Cmd does not. What's else there? I have Bash here but that's
>> a Cygwin executable. And Cygwin Python does not work like Windows
>> Python.

> Since subprocess.Popen doesn't call ShellExecuteEx, it would make no
> sense for it to use a custom search that tries appending all of the
> file extensions in %PathExt%. Thus the only case for Popen to
> implement its own search is to avoid the implicit directories that
> CreateProcess searches ahead of %Path%. But implicitly searching the
> application directory and system directories ahead of %Path% is a
> feature for Windows applications, and Python is primarily an
> application development language on Windows -- not a system
> administration language that takes the place of the shell.

Due to backwards compatibility this is unlikely to change.  However, if you can make the case that this missing functionality is important then perhaps one more parameter can be specified to Popen, or a new command added to subprocess, that deals with it.

See https://mail.python.org/pipermail/python-ideas/2016-November/043620.html for some good ideas on what would be needed for such a proposal.

--
~Ethan~



From triccare at gmail.com  Fri Nov 11 11:26:58 2016
From: triccare at gmail.com (triccare triccare)
Date: Fri, 11 Nov 2016 11:26:58 -0500
Subject: Why keys method does not work with MutableMapping?
Message-ID: <CADtmbDByGOy0=0TSDtDH9VSq67fkz5FK4r_jZrZHcc+fhXanAA@mail.gmail.com>

I have a class that completely implements MutableMapping, meaning that all
the abstract methods are implemented. However, the keys method no longer
returns the keys, but simply a repr of the instance.  Example is below.
Same is true for the items method.

It would seem that, if all the abstract methods have been implemented, the
keys and items methods should be able to perform exactly like the native
dict versions. There does not seem to be a need to override these methods.

Thank you for your time.
triccare

Code:

from collections import MutableMapping

class MyDict(MutableMapping):

    def __init__(self, *args, **kwargs):
        self.data = dict(*args, **kwargs)

    def __getitem__(self, key):
        return self.data[self.__keytransform__(key)]

    def __setitem__(self, key, value):
        self.data[self.__keytransform__(key)] = value

    def __delitem__(self, key):
        del self.data[self.__keytransform__(key)]

    def __iter__(self):
        return iter(self.data)

    def __len__(self):
        return len(self.data)

    def __keytransform__(self, key):
        return key

md = MyDict({'a': 1, 'b':2})
md.keys()
==> KeysView(<keys.MyDict object at 0x105f7f7b8>)


From triccare at gmail.com  Fri Nov 11 12:27:08 2016
From: triccare at gmail.com (triccare triccare)
Date: Fri, 11 Nov 2016 12:27:08 -0500
Subject: Why keys method does not work with MutableMapping?
Message-ID: <CADtmbDA9tDaq8BHLhfx+K4e_v6ifSnQ1h2KOzVqJygkF3eN6mQ@mail.gmail.com>

Greetings,

Apologies if this has shown up twice; I jumped the gun sending before
confirming registration.

I have a class that completely implements MutableMapping, meaning that all
the abstract methods are implemented. However, the keys method no longer
returns the keys, but simply a repr of the instance.  Example is below.
Same is true for the items method.

It would seem that, if all the abstract methods have been implemented, the
keys and items methods should be able to perform exactly like the native
dict versions. There does not seem to be a need to override these methods.

Thank you for your time.
triccare

Code:

from collections import MutableMapping

class MyDict(MutableMapping):

    def __init__(self, *args, **kwargs):
        self.data = dict(*args, **kwargs)

    def __getitem__(self, key):
        return self.data[self.__keytransform__(key)]

    def __setitem__(self, key, value):
        self.data[self.__keytransform__(key)] = value

    def __delitem__(self, key):
        del self.data[self.__keytransform__(key)]

    def __iter__(self):
        return iter(self.data)

    def __len__(self):
        return len(self.data)

    def __keytransform__(self, key):
        return key

md = MyDict({'a': 1, 'b':2})
md.keys()
==> KeysView(<keys.MyDict object at 0x105f7f7b8>)


From rgaddi at highlandtechnology.invalid  Fri Nov 11 13:03:40 2016
From: rgaddi at highlandtechnology.invalid (Rob Gaddi)
Date: Fri, 11 Nov 2016 18:03:40 -0000 (UTC)
Subject: Why keys method does not work with MutableMapping?
References: <CADtmbDA9tDaq8BHLhfx+K4e_v6ifSnQ1h2KOzVqJygkF3eN6mQ@mail.gmail.com>
 <mailman.107.1478885231.30068.python-list@python.org>
Message-ID: <o0515r$2du$1@dont-email.me>

triccare triccare wrote:

> Greetings,
>
> Apologies if this has shown up twice; I jumped the gun sending before
> confirming registration.
>
> I have a class that completely implements MutableMapping, meaning that all
> the abstract methods are implemented. However, the keys method no longer
> returns the keys, but simply a repr of the instance.  Example is below.
> Same is true for the items method.
>
> It would seem that, if all the abstract methods have been implemented, the
> keys and items methods should be able to perform exactly like the native
> dict versions. There does not seem to be a need to override these methods.
>
> Thank you for your time.
> triccare
>
> Code:
>
> from collections import MutableMapping
>
> class MyDict(MutableMapping):
>
>     def __init__(self, *args, **kwargs):
>         self.data = dict(*args, **kwargs)
>
>     def __getitem__(self, key):
>         return self.data[self.__keytransform__(key)]
>
>     def __setitem__(self, key, value):
>         self.data[self.__keytransform__(key)] = value
>
>     def __delitem__(self, key):
>         del self.data[self.__keytransform__(key)]
>
>     def __iter__(self):
>         return iter(self.data)
>
>     def __len__(self):
>         return len(self.data)
>
>     def __keytransform__(self, key):
>         return key
>
> md = MyDict({'a': 1, 'b':2})
> md.keys()
> ==> KeysView(<keys.MyDict object at 0x105f7f7b8>)

Nope, that's exactly right.  That's the python3 behavior.

>>> d = {'a': 1, 'b':2}
>>> d.keys()
dict_keys(['b', 'a'])

Keys returns a dedicated keys object now, not just a list.  That thing
you got back isn't a repr string; it's the actual object.  If it were a
string it'd be quoted.

Try list(md.keys()).


-- 
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order.  See above to fix.


From python.list at tim.thechases.com  Fri Nov 11 13:34:33 2016
From: python.list at tim.thechases.com (Tim Chase)
Date: Fri, 11 Nov 2016 12:34:33 -0600
Subject: update certain key-value pairs of a dict from another dict
In-Reply-To: <o04djd$g9o$1@blaine.gmane.org>
References: <CAF_EDkuib3bNhu2ufeOosEpUH=b6RRsnjD6OGWFkyxrrB=_tSg@mail.gmail.com>
 <20161111055917.100de01c@bigbox.christie.dr>
 <o04djd$g9o$1@blaine.gmane.org>
Message-ID: <20161111123433.6dce6ca7@bigbox.christie.dr>

On 2016-11-11 13:29, Peter Otten wrote:
> The same using update(), with a generator expression that avoids
> the intermediate dict:
> 
> >>> dict1 = {'A': 'a', 'B': 'b', 'C': 'c'}
> >>> dict1.update((k, dict2[k]) for k in desired & dict1.keys() & 
> dict2.keys())

Huh.  Handy to file that new knowledge away.  I'd not realized it
could take a 2-tuple iterable, but my previous example would then be
more cleanly written as

  dict1.update((k,v) for k,v in dict.items() if k in desired)

But yes, certainly a couple edge cases depending on the dict
sizes, the size of the "desired" set, and what should happen in the
event dict2 (or "desired") has keys that dict1 doesn't.

-tkc





From fcracker79 at gmail.com  Fri Nov 11 14:42:45 2016
From: fcracker79 at gmail.com (mirko bonasorte)
Date: Fri, 11 Nov 2016 11:42:45 -0800 (PST)
Subject: Promoting your own library
Message-ID: <66301f14-ac21-4fb8-8405-0af54f9b8ffa@googlegroups.com>

Hi all,

what is the most appropriate way for a developer to promote his own Python library? I mean, apart from deploying it in Pypi and making the source code available in a SCV repository...

Thanks.

Mirko


From jelena.tavcar at gmail.com  Fri Nov 11 15:36:10 2016
From: jelena.tavcar at gmail.com (Jelena Tavcar)
Date: Fri, 11 Nov 2016 21:36:10 +0100
Subject: Fwd: Python does not start
In-Reply-To: <CAKgG=gyOOca0Ea_W7GfWUY_7dHyYHWr51bvWwScXn=uj9nLa5w@mail.gmail.com>
References: <CAKgG=gyOOca0Ea_W7GfWUY_7dHyYHWr51bvWwScXn=uj9nLa5w@mail.gmail.com>
Message-ID: <CAKgG=gw+gsE+jyG_6x7Ts-wRh67CJthB3WKuwzhvtZAE6NXwKw@mail.gmail.com>

Dear Sirs,
when I click on the icon "IDLE (Python 3.5 32-bit)" nothing happens. It
used to work and at some point it just didn't react. I have reinstalled the
program several times and no improvement. I have also tried to run it as
administrator, and installed other version too, but neither of them worked.
I have Windows 10. What do I do? Would you please suggest how to resolve
this issue?
Thanks in advance,
kind regards,
Jelena


From daniele.bucciero at outlook.com  Fri Nov 11 15:47:48 2016
From: daniele.bucciero at outlook.com (Daniele Bucciero)
Date: Fri, 11 Nov 2016 20:47:48 +0000
Subject: Python does not start
In-Reply-To: <CAKgG=gw+gsE+jyG_6x7Ts-wRh67CJthB3WKuwzhvtZAE6NXwKw@mail.gmail.com>
References: <CAKgG=gyOOca0Ea_W7GfWUY_7dHyYHWr51bvWwScXn=uj9nLa5w@mail.gmail.com>
 <CAKgG=gw+gsE+jyG_6x7Ts-wRh67CJthB3WKuwzhvtZAE6NXwKw@mail.gmail.com>
Message-ID: <LO1P123MB102516B1FDD892FCE769803C9ABB0@LO1P123MB1025.GBRP123.PROD.OUTLOOK.COM>

Did you check your Windows Event Viewer?

Did you notice something wrong in Application Events?

Regards
Daniele Bucciero

-----Original Message-----
From: Python-list
[mailto:python-list-bounces+daniele.bucciero=outlook.com at python.org] On
Behalf Of Jelena Tavcar
Sent: venerd? 11 novembre 2016 21:36
To: python-list at python.org
Subject: Fwd: Python does not start

Dear Sirs,
when I click on the icon "IDLE (Python 3.5 32-bit)" nothing happens. It used
to work and at some point it just didn't react. I have reinstalled the
program several times and no improvement. I have also tried to run it as
administrator, and installed other version too, but neither of them worked.
I have Windows 10. What do I do? Would you please suggest how to resolve
this issue?
Thanks in advance,
kind regards,
Jelena
--
https://mail.python.org/mailman/listinfo/python-list


From eryksun at gmail.com  Fri Nov 11 15:59:54 2016
From: eryksun at gmail.com (eryk sun)
Date: Fri, 11 Nov 2016 20:59:54 +0000
Subject: Python does not start
In-Reply-To: <CAKgG=gw+gsE+jyG_6x7Ts-wRh67CJthB3WKuwzhvtZAE6NXwKw@mail.gmail.com>
References: <CAKgG=gyOOca0Ea_W7GfWUY_7dHyYHWr51bvWwScXn=uj9nLa5w@mail.gmail.com>
 <CAKgG=gw+gsE+jyG_6x7Ts-wRh67CJthB3WKuwzhvtZAE6NXwKw@mail.gmail.com>
Message-ID: <CACL+1at+o1bcgRnNKpGCbntjq4=-j=qnbLtpXmGyCABfnjckfQ@mail.gmail.com>

On Fri, Nov 11, 2016 at 8:36 PM, Jelena Tavcar <jelena.tavcar at gmail.com> wrote:
>
> when I click on the icon "IDLE (Python 3.5 32-bit)" nothing happens. It
> used to work and at some point it just didn't react.

Try running IDLE from a command prompt by entering the following command:

    py -3.5-32 -m idlelib

If this prints an error, please provide the entire error message
(copy/paste), including the complete traceback. Do not attach a
screenshot; we won't get it. python-list is text only.


From tjreedy at udel.edu  Fri Nov 11 17:13:26 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 11 Nov 2016 17:13:26 -0500
Subject: Python does not start
In-Reply-To: <CACL+1at+o1bcgRnNKpGCbntjq4=-j=qnbLtpXmGyCABfnjckfQ@mail.gmail.com>
References: <CAKgG=gyOOca0Ea_W7GfWUY_7dHyYHWr51bvWwScXn=uj9nLa5w@mail.gmail.com>
 <CAKgG=gw+gsE+jyG_6x7Ts-wRh67CJthB3WKuwzhvtZAE6NXwKw@mail.gmail.com>
 <CACL+1at+o1bcgRnNKpGCbntjq4=-j=qnbLtpXmGyCABfnjckfQ@mail.gmail.com>
Message-ID: <o05fq4$9jp$1@blaine.gmane.org>

On 11/11/2016 3:59 PM, eryk sun wrote:
> On Fri, Nov 11, 2016 at 8:36 PM, Jelena Tavcar <jelena.tavcar at gmail.com> wrote:
>>
>> when I click on the icon "IDLE (Python 3.5 32-bit)" nothing happens. It

Does python itself work?  If so, which micro version.  It is in the 
startup line that begins 'Python 3.5.2+ ...'.  There was bug in 3.5.0 
that was fixed in 3.5.1 which could impact this issue.  (If you are not 
using 3.5.2, I recommend upgrading.  It is very easy.)

>> used to work and at some point it just didn't react.

That means that something changed on your system, but what?  Since you 
re-installed 'the program', by which I presume you mean python3.5, the 
Python and IDLE files should not be the problem.  I would otherwise 
suggest the user config files, usually in directory 
C:/Users/<account>/.idlerc/, where <account> is the account name.  But 
since you tried both an admin and user account, that seems unlikey.

> Try running IDLE from a command prompt by entering the following command:
>
>     py -3.5-32 -m idlelib
>
> If this prints an error, please provide the entire error message
> (copy/paste), including the complete traceback. Do not attach a
> screenshot; we won't get it. python-list is text only.

Definitely do this.  A third possibility is that you saved a file with 
the same name as an stdlib file in a place where it gets imported by 
IDLE instead of the stdlib module.  An error message should give a hint.

-- 
Terry Jan Reedy



From subhabangalore at gmail.com  Fri Nov 11 17:29:45 2016
From: subhabangalore at gmail.com (subhabangalore at gmail.com)
Date: Fri, 11 Nov 2016 14:29:45 -0800 (PST)
Subject: Python String Handling
Message-ID: <876a470a-a535-4450-985c-69b6f5b46f17@googlegroups.com>

I have a string 
"Hello my name is Richard"

I have a list of words as,
['Hello/Hi','my','name','is','Richard/P']

I want to identify the match of 'Hello' and 'Richard'
in list, and replace them with 'Hello/Hi" and 'Richard/P'
respectively.

The result should look like,
"Hello/Hi my name is Richard/P".

Simple replace method may not work.

I was trying the following script. 


import fuzzywuzzy
from fuzzywuzzy import fuzz
from fuzzywuzzy import process
import itertools
def sometry():
     x1="Hello my name is Richard"
     x2=x1.split()
     x3=['Hello/Hi','my','name','is','Richard/P']
     list1=[]
     for i in x2:
          x4=process.extractOne(i, x3)
          print x4
          x5=x4[0]
          print x5
          x6=[x5 if x==i else x for x in x2]
          print x6
          list1.append(x6)

     b1=list1
     print b1
     merged = list(itertools.chain.from_iterable(b1))
     merged1=list(set(merged))
     print merged1

I am working in Python2.x on MS-Windows. 
This is a simple practice script so I have not followed style guides.
Apology for any indentation error.

I am trying if any one of the members may give any idea how may I achieve it.

Thanks in Advance. 


From tomuxiong at gmx.com  Fri Nov 11 17:44:24 2016
From: tomuxiong at gmx.com (Thomas Nyberg)
Date: Fri, 11 Nov 2016 17:44:24 -0500
Subject: Python String Handling
In-Reply-To: <876a470a-a535-4450-985c-69b6f5b46f17@googlegroups.com>
References: <876a470a-a535-4450-985c-69b6f5b46f17@googlegroups.com>
Message-ID: <2046f25c-fa98-ac8c-ac7c-d81bf6263b15@gmx.com>

On 11/11/2016 05:29 PM, subhabangalore at gmail.com wrote:
> I have a string
> "Hello my name is Richard"
>
> I want to identify the match of 'Hello' and 'Richard'
> in list, and replace them with 'Hello/Hi" and 'Richard/P'
> respectively.
>
> The result should look like,
> "Hello/Hi my name is Richard/P".
>
> Simple replace method may not work.
>
Why would simple replace not work? I.e. what's wrong with this?

s = '"Hello my name is Richard"'
s.replace("Hello", "Hello/Hi").replace("Richard", "Richard/P")

I mean maybe you can't use that, but if you can't use that, then I'm 
probably not understanding your question.

Cheers,
Thomas


From eryksun at gmail.com  Fri Nov 11 18:10:45 2016
From: eryksun at gmail.com (eryk sun)
Date: Fri, 11 Nov 2016 23:10:45 +0000
Subject: Python does not start
In-Reply-To: <58263a5b.2636c20a.6c87.4e50@mx.google.com>
References: <CAKgG=gyOOca0Ea_W7GfWUY_7dHyYHWr51bvWwScXn=uj9nLa5w@mail.gmail.com>
 <CAKgG=gw+gsE+jyG_6x7Ts-wRh67CJthB3WKuwzhvtZAE6NXwKw@mail.gmail.com>
 <CACL+1at+o1bcgRnNKpGCbntjq4=-j=qnbLtpXmGyCABfnjckfQ@mail.gmail.com>
 <58263a5b.2636c20a.6c87.4e50@mx.google.com>
Message-ID: <CACL+1auObE6M_1jGu2Jn9dU259e8ZgZV4Q_+1wVJgJCSdenFeg@mail.gmail.com>

On Fri, Nov 11, 2016 at 9:38 PM,  <jelena.tavcar at gmail.com> wrote:
> It says that ?py-3.5-32-m? is not recognized as an internal or external
> command, operable program or batch file.

There's supposed to be a space after "py". Let's add the .exe
extension to make this clearer:

    py.exe -3.5-32 -m idlelib

And please do not paraphrase error messages. Copy and paste errors
verbatim. Also, make sure to reply-all to include python-list.


From ben+python at benfinney.id.au  Fri Nov 11 18:59:15 2016
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sat, 12 Nov 2016 10:59:15 +1100
Subject: Promoting your own library
References: <66301f14-ac21-4fb8-8405-0af54f9b8ffa@googlegroups.com>
Message-ID: <8537ix7elo.fsf@benfinney.id.au>

mirko bonasorte <fcracker79 at gmail.com> writes:

> what is the most appropriate way for a developer to promote his own
> Python library?

The general answer is: Publish it with full metadata on PyPI. That's
where the Python community looks to find third-party modules, so that's
the place to put it.

Beyond that? You'll need to know whom you want to promote *to*. Find out
where they get their information, and participate respectfully in those
forums.

-- 
 \        ?Sane people have an appropriate perspective on the relative |
  `\     importance of foodstuffs and human beings. Crazy people can't |
_o__)                 tell the difference.? ?Paul Z. Myers, 2010-04-18 |
Ben Finney



From steve+python at pearwood.info  Fri Nov 11 20:41:10 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sat, 12 Nov 2016 12:41:10 +1100
Subject: Python String Handling
References: <876a470a-a535-4450-985c-69b6f5b46f17@googlegroups.com>
Message-ID: <58267339$0$1617$c3e8da3$5496439d@news.astraweb.com>

On Sat, 12 Nov 2016 09:29 am, subhabangalore at gmail.com wrote:

> I have a string
> "Hello my name is Richard"
> 
> I have a list of words as,
> ['Hello/Hi','my','name','is','Richard/P']
> 
> I want to identify the match of 'Hello' and 'Richard'
> in list, and replace them with 'Hello/Hi" and 'Richard/P'
> respectively.
> 
> The result should look like,
> "Hello/Hi my name is Richard/P".

Looks like you want:


mystring = "Hello my name is Richard"
words = ['Hello/Hi', 'my', 'name', 'is', 'Richard/P']
result = " ".join(words)

assert result == "Hello/Hi my name is Richard/P"


and mystring is irrelevant.




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From subhabangalore at gmail.com  Sat Nov 12 03:46:58 2016
From: subhabangalore at gmail.com (subhabangalore at gmail.com)
Date: Sat, 12 Nov 2016 00:46:58 -0800 (PST)
Subject: Python String Handling
In-Reply-To: <58267339$0$1617$c3e8da3$5496439d@news.astraweb.com>
References: <876a470a-a535-4450-985c-69b6f5b46f17@googlegroups.com>
 <58267339$0$1617$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <3c137163-25d7-47db-af1a-d5522af58188@googlegroups.com>

On Saturday, November 12, 2016 at 7:34:31 AM UTC+5:30, Steve D'Aprano wrote:
> On Sat, 12 Nov 2016 09:29 am wrote:
> 
> > I have a string
> > "Hello my name is Richard"
> > 
> > I have a list of words as,
> > ['Hello/Hi','my','name','is','Richard/P']
> > 
> > I want to identify the match of 'Hello' and 'Richard'
> > in list, and replace them with 'Hello/Hi" and 'Richard/P'
> > respectively.
> > 
> > The result should look like,
> > "Hello/Hi my name is Richard/P".
> 
> Looks like you want:
> 
> 
> mystring = "Hello my name is Richard"
> words = ['Hello/Hi', 'my', 'name', 'is', 'Richard/P']
> result = " ".join(words)
> 
> assert result == "Hello/Hi my name is Richard/P"
> 
> 
> and mystring is irrelevant.
> 
> 
> 
> 
> -- 
> Steve
> ?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
> enough, things got worse.

Thank you all for your kind time. 
The problem is slightly more complex.

I am restating the problem. 

"Hello my name is Richard"

is a string. 

I have tagged the words Hello and Richard
as "Hello/Hi" and "Richard/P". 
After this I could get the string as a list of words
as in,
['Hello/Hi','my','name','is','Richard/P'] 

Now I want to replace the string with 
Hello/Hi my name is Richard/P

It may seem a joining of list but is not because
if I try to make, 
['Hello/Hi','my/M','name','is/I','Richard/P'] 

I may do, but doing the following string 

Hello/Hi my/M<HM> name is/I Richard/P<IP>

is tough as entities with tag may vary. I have to make
a rule. 

I am trying to recognize the index of the word in
the list, pop it and replace with new value and joining
the list as string.

This is okay but as expert people if you have any smarter
suggestion.

Thanks in advance

 




From jelena.tavcar at gmail.com  Sat Nov 12 04:13:45 2016
From: jelena.tavcar at gmail.com (jelena.tavcar at gmail.com)
Date: Sat, 12 Nov 2016 10:13:45 +0100
Subject: Python does not start
In-Reply-To: <CACL+1auObE6M_1jGu2Jn9dU259e8ZgZV4Q_+1wVJgJCSdenFeg@mail.gmail.com>
References: <CAKgG=gyOOca0Ea_W7GfWUY_7dHyYHWr51bvWwScXn=uj9nLa5w@mail.gmail.com>
 <CAKgG=gw+gsE+jyG_6x7Ts-wRh67CJthB3WKuwzhvtZAE6NXwKw@mail.gmail.com>
 <CACL+1at+o1bcgRnNKpGCbntjq4=-j=qnbLtpXmGyCABfnjckfQ@mail.gmail.com>
 <58263a5b.2636c20a.6c87.4e50@mx.google.com>
 <CACL+1auObE6M_1jGu2Jn9dU259e8ZgZV4Q_+1wVJgJCSdenFeg@mail.gmail.com>
Message-ID: <5826dd49.0d8d1c0a.3fe4e.32d2@mx.google.com>

I?m sending you the error message from command prompt

C:\Users\Jelena>py.exe -3.5-32 -m idlelib
Traceback (most recent call last):
  File "C:\Users\Jelena\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\Jelena\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\Jelena\AppData\Local\Programs\Python\Python35-32\lib\idlelib\__main__.py", line 7, in <module>
    idlelib.PyShell.main()
  File "C:\Users\Jelena\AppData\Local\Programs\Python\Python35-32\lib\idlelib\PyShell.py", line 1582, in main
    shell = flist.open_shell()
  File "C:\Users\Jelena\AppData\Local\Programs\Python\Python35-32\lib\idlelib\PyShell.py", line 320, in open_shell
    self.pyshell = PyShell(self)
  File "C:\Users\Jelena\AppData\Local\Programs\Python\Python35-32\lib\idlelib\PyShell.py", line 867, in __init__
    OutputWindow.__init__(self, flist, None, None)
  File "C:\Users\Jelena\AppData\Local\Programs\Python\Python35-32\lib\idlelib\OutputWindow.py", line 16, in __init__
    EditorWindow.__init__(self, *args)
  File "C:\Users\Jelena\AppData\Local\Programs\Python\Python35-32\lib\idlelib\EditorWindow.py", line 172, in __init__
    self.apply_bindings()
  File "C:\Users\Jelena\AppData\Local\Programs\Python\Python35-32\lib\idlelib\EditorWindow.py", line 1068, in apply_bindings
    text.event_add(event, *keylist)
  File "C:\Users\Jelena\AppData\Local\Programs\Python\Python35-32\lib\idlelib\MultiCall.py", line 374, in event_add
    widget.event_add(self, virtual, seq)
  File "C:\Users\Jelena\AppData\Local\Programs\Python\Python35-32\lib\tkinter\__init__.py", line 1504, in event_add
    self.tk.call(args)
_tkinter.TclError: bad event type or keysym "Alt"

Sent from Mail for Windows 10

From: eryk sun
Sent: 12 November 2016 00:11
To: python-list at python.org
Cc: jelena.tavcar at gmail.com
Subject: Re: Python does not start

On Fri, Nov 11, 2016 at 9:38 PM,  <jelena.tavcar at gmail.com> wrote:
> It says that ?py-3.5-32-m? is not recognized as an internal or external
> command, operable program or batch file.

There's supposed to be a space after "py". Let's add the .exe
extension to make this clearer:

    py.exe -3.5-32 -m idlelib

And please do not paraphrase error messages. Copy and paste errors
verbatim. Also, make sure to reply-all to include python-list.



From dieter at handshake.de  Sat Nov 12 04:13:52 2016
From: dieter at handshake.de (dieter)
Date: Sat, 12 Nov 2016 10:13:52 +0100
Subject: Reporting a Bug
References: <CAG7NVB=XCwtmvvrCEKYMJPjbvpXRrFDOiJ7Om85--8CE1YDmJA@mail.gmail.com>
Message-ID: <87vavtdprj.fsf@handshake.de>

Vidyalakshmi Rao <vivabs at gmail.com> writes:
> ...
> *Issue: Spyder hangs in instances while iterating over a list containing
> rows with nothing in it.*
>
>
>     for eachinfo in range(len(textlist)): #remove non-ASCII characters from
> the textlist
>         textlist[eachinfo] = re.sub(r'[^\x00-\x7F]+','', textlist[eachinfo])
>
> This results in a list *textlist*, which contains some rows with nothing in
> it.

With "row with nothing in it", you mean "row containing the empty string"?


> When such a list is iterated over (as below), Spyder hangs indefinitely.
>
>     for item in textlist:
>         print item
>
> While debugging through Spyder (Ctrl+F5), it hangs indefinitely. But, if we
> just run(F5), executes without a glitch.
> Am i doing something wrong or is it a bug? Is it related to Spyder?
> I am a novice to Python. Please direct me if this is not a Python issue but
> an Anaconda issue.

I do not know "Sypder" (nor "Anaconda"). However, your observation does not
indicate a Python problem. For a Python problem, the "run(F5)" should
misbehave as well.

Maybe, the question will get better answers on an "Anaconda" mailing list?



From vek.m1234 at gmail.com  Sat Nov 12 07:07:34 2016
From: vek.m1234 at gmail.com (Veek M)
Date: Sat, 12 Nov 2016 17:37:34 +0530
Subject: Web Scraping
References: <e5df1d3f-6593-4dff-b9ab-de0691839c54@googlegroups.com>
Message-ID: <o070l9$o8v$1@dont-email.me>

121sukha wrote:

> I am new to python and I want to use web scraping to download songs
> from website. how do I write code to check if the website has uploaded
> a new song and have that song automatically be downloaded onto my
> computer. I know how to use the requests.get() module but i am more
> interested in knowing how to download and save every new song that the
> website uploads on the site. I would extremely appreciate the help
> from anyone. Thanks!

What you're doing is illegal but Youtube/websites have a lot of stuff up 
that's easily accessible - the moral issue seems ambiguous especially if 
you use adblocker.. 

Check for a RSS feed (orange icon thingy) or maybe hash (md5) the page 
contents(html) and compare against your previous run? Also check out the 
etree module for extracting links and other useful data reliably.


From guy.asor132 at gmail.com  Sat Nov 12 07:58:20 2016
From: guy.asor132 at gmail.com (guy asor)
Date: Sat, 12 Nov 2016 04:58:20 -0800 (PST)
Subject: how to print variable few time?
Message-ID: <7b3d77cb-c42d-48bb-876f-ef36d9dc27f4@googlegroups.com>

hello!

this is my code:

word=raw_input()
print word*3


with this code im getting - wordwordword.
what changes i need to make to get - word word word - instead?

thanks


From jon+usenet at unequivocal.eu  Sat Nov 12 08:02:02 2016
From: jon+usenet at unequivocal.eu (Jon Ribbens)
Date: Sat, 12 Nov 2016 13:02:02 -0000 (UTC)
Subject: Python String Handling
References: <876a470a-a535-4450-985c-69b6f5b46f17@googlegroups.com>
 <58267339$0$1617$c3e8da3$5496439d@news.astraweb.com>
 <3c137163-25d7-47db-af1a-d5522af58188@googlegroups.com>
Message-ID: <slrno2e4n9.2bv.jon+usenet@sable.unequivocal.eu>

On 2016-11-12, subhabangalore at gmail.com <subhabangalore at gmail.com> wrote:
> I am restating the problem. 
>
> "Hello my name is Richard"
>
> is a string. 
>
> I have tagged the words Hello and Richard
> as "Hello/Hi" and "Richard/P". 
> After this I could get the string as a list of words
> as in,
> ['Hello/Hi','my','name','is','Richard/P'] 
>
> Now I want to replace the string with 
> Hello/Hi my name is Richard/P
>
> It may seem a joining of list but is not because
> if I try to make, 
> ['Hello/Hi','my/M','name','is/I','Richard/P'] 
>
> I may do, but doing the following string 
>
> Hello/Hi my/M<HM> name is/I Richard/P<IP>
>
> is tough as entities with tag may vary. I have to make
> a rule. 
>
> I am trying to recognize the index of the word in
> the list, pop it and replace with new value and joining
> the list as string.

I'm afraid your description of the problem is still rather... opaque.
>From your original post it looked like the answer would be:

    def replacer(target, replacements):
        for replacement in replacements:
            if "/" in replacement:
                target = target.replace(*replacement.split("/", 1))
        return target

    print(replacer("Hello my name is Richard",
                   ['Hello/Hi','my','name','is','Richard/P']))

but in your new description where you start talking about word indices
I have no idea if that's what you actually wanted.


From ben.usenet at bsb.me.uk  Sat Nov 12 08:30:17 2016
From: ben.usenet at bsb.me.uk (Ben Bacarisse)
Date: Sat, 12 Nov 2016 13:30:17 +0000
Subject: how to print variable few time?
References: <7b3d77cb-c42d-48bb-876f-ef36d9dc27f4@googlegroups.com>
Message-ID: <87r36gn7va.fsf@bsb.me.uk>

guy asor <guy.asor132 at gmail.com> writes:

> this is my code:
>
> word=raw_input()
> print word*3
>
>
> with this code im getting - wordwordword.
> what changes i need to make to get - word word word - instead?

Two hints:

What does ['a']*3 give you?

What about '/'.join(['a', 'b', 'c'])?

Can you find a way now?

-- 
Ben.


From steve+python at pearwood.info  Sat Nov 12 08:55:56 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sun, 13 Nov 2016 00:55:56 +1100
Subject: how to print variable few time?
References: <7b3d77cb-c42d-48bb-876f-ef36d9dc27f4@googlegroups.com>
Message-ID: <58271f6f$0$1591$c3e8da3$5496439d@news.astraweb.com>

On Sat, 12 Nov 2016 11:58 pm, guy asor wrote:

> hello!
> 
> this is my code:
> 
> word=raw_input()
> print word*3
> 
> 
> with this code im getting - wordwordword.
> what changes i need to make to get - word word word - instead?

Lots of ways! Here is the simplest, but longest. Call print three times,
being careful to not allow a newline to be printed until the end:

print word,  # note the comma, that suppresses the new line
print word,
print word  # no comma


What if you wanted to print it fifty times? Use a loop.


for i in range(50):
    print word,  # note the comma
# At the end of the loop, print a blank to get a new line.
print



Here's a nice trick that's a bit more mysterious for beginners, but works
well:


print ' '.join([word]*3)


If you don't understand it, break it up into each piece:


[word]*3 creates a list [word, word, word]

' '.join(the list) makes a string with the words separated by spaces:

' '.join(['quick', 'brown', 'fox']) => 'quick brown fox'


and then you finally print the result.



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Sat Nov 12 09:09:32 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sun, 13 Nov 2016 01:09:32 +1100
Subject: Web Scraping
References: <e5df1d3f-6593-4dff-b9ab-de0691839c54@googlegroups.com>
 <o070l9$o8v$1@dont-email.me>
Message-ID: <5827229d$0$1600$c3e8da3$5496439d@news.astraweb.com>

On Sat, 12 Nov 2016 11:07 pm, Veek M wrote:

> 121sukha wrote:
> 
>> I am new to python and I want to use web scraping to download songs
>> from website. how do I write code to check if the website has uploaded
>> a new song and have that song automatically be downloaded onto my
>> computer. I know how to use the requests.get() module but i am more
>> interested in knowing how to download and save every new song that the
>> website uploads on the site. I would extremely appreciate the help
>> from anyone. Thanks!
> 
> What you're doing is illegal 

Really? Everywhere in the world? And you're sure of this because you're a
lawyer?

Even if the website uses public domain or freely licenced songs?

Even if the downloads count as fair use, time-shifting or format-shifting?



> but Youtube/websites have a lot of stuff up 
> that's easily accessible - the moral issue seems ambiguous especially if
> you use adblocker..

Doesn't seem ambiguous to me. 

Or are you one of those people who think that you are a thief for leaving
the room when ads are playing on TV?

https://yro.slashdot.org/story/02/05/02/0550214/turner-ceo-pvr-users-are-thieves


Downloading may, or may not, violate the terms of use of the website. But
you *literally* cannot watch the video without downloading it: in order for
the video to play in your browser, it must be downloaded. That's the end of
the story. "Streaming video" is just another way of saying "downloading
video, where the video is deleted afterwards".

(Actually, streaming video is just another way of saying "lots of pauses,
stuttering, dropped frames and a really awful user experience".)



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From esj at harvee.org  Sat Nov 12 09:29:45 2016
From: esj at harvee.org (Eric S. Johansson)
Date: Sat, 12 Nov 2016 09:29:45 -0500
Subject: Promoting your own library
In-Reply-To: <8537ix7elo.fsf@benfinney.id.au>
References: <66301f14-ac21-4fb8-8405-0af54f9b8ffa@googlegroups.com>
 <8537ix7elo.fsf@benfinney.id.au>
Message-ID: <895cf749-0a54-100f-6cc1-92da4aaa6d86@harvee.org>



On 11/11/2016 6:59 PM, Ben Finney wrote:
> mirko bonasorte <fcracker79 at gmail.com> writes:
>
>> what is the most appropriate way for a developer to promote his own
>> Python library?
> The general answer is: Publish it with full metadata on PyPI. That's
> where the Python community looks to find third-party modules, so that's
> the place to put it.
>
> Beyond that? You'll need to know whom you want to promote *to*. Find out
> where they get their information, and participate respectfully in those
> forums.
>
I could also use some of this information. I want to publish a couple of
things that I found useful. My creations tend to be single file modules
or commands and what I hope to understand from your guidance is how to
bundle that single file module or standalone program for publication.

--- eric





From vek.m1234 at gmail.com  Sat Nov 12 10:18:45 2016
From: vek.m1234 at gmail.com (Veek M)
Date: Sat, 12 Nov 2016 20:48:45 +0530
Subject: Web Scraping
References: <e5df1d3f-6593-4dff-b9ab-de0691839c54@googlegroups.com>
 <o070l9$o8v$1@dont-email.me>
 <5827229d$0$1600$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o07brp$u0b$1@dont-email.me>

Steve D'Aprano wrote:

> On Sat, 12 Nov 2016 11:07 pm, Veek M wrote:
> 
>> 121sukha wrote:
>> 
>>> I am new to python and I want to use web scraping to download songs
>>> from website. how do I write code to check if the website has
>>> uploaded a new song and have that song automatically be downloaded
>>> onto my computer. I know how to use the requests.get() module but i
>>> am more interested in knowing how to download and save every new
>>> song that the website uploads on the site. I would extremely
>>> appreciate the help from anyone. Thanks!
>> 
>> What you're doing is illegal
> 
> Really? Everywhere in the world? And you're sure of this because
> you're a lawyer?

'probably illegal' (i did mentally assume he's from India - the nick is 
very indianish and since I am Indian and noticed many badly formatted 
posts I felt obliged to help out)

> Even if the website uses public domain or freely licenced songs?
> 
> Even if the downloads count as fair use, time-shifting or
> format-shifting?

ah true

>> but Youtube/websites have a lot of stuff up
>> that's easily accessible - the moral issue seems ambiguous especially
>> if you use adblocker..
> 
> Doesn't seem ambiguous to me.

> Or are you one of those people who think that you are a thief for
> leaving the room when ads are playing on TV?
> 
> https://yro.slashdot.org/story/02/05/02/0550214/turner-ceo-pvr-users-are-thieves

yep as in.. do the networks have a right to tempt me
 
> 
> Downloading may, or may not, violate the terms of use of the website.
> But you *literally* cannot watch the video without downloading it: in
> order for the video to play in your browser, it must be downloaded.
> That's the end of the story. "Streaming video" is just another way of
> saying "downloading video, where the video is deleted afterwards".
> 
> (Actually, streaming video is just another way of saying "lots of
> pauses, stuttering, dropped frames and a really awful user
> experience".)

also true, but gramophone records were an inconvenience we put up with 
lacking a better solution.. so an inconvenience strictly speaking 
doesn't imply you can break the law.. however, is it fair to tempt us 
with these delights.. hence my moral confusion..

Anyway.. yes I agree that I have no business telling people off (and 
that was not my intention) it was a succinct warning of the dangers.. 


From gvm2121 at gmail.com  Sat Nov 12 12:48:39 2016
From: gvm2121 at gmail.com (Gonzalo V)
Date: Sat, 12 Nov 2016 14:48:39 -0300
Subject: Web Scraping
In-Reply-To: <o07brp$u0b$1@dont-email.me>
References: <e5df1d3f-6593-4dff-b9ab-de0691839c54@googlegroups.com>
 <o070l9$o8v$1@dont-email.me>
 <5827229d$0$1600$c3e8da3$5496439d@news.astraweb.com>
 <o07brp$u0b$1@dont-email.me>
Message-ID: <CAF40P_3msdOj=KXHYdPfQXxAZ+AXV1iOnX8GgRAuaVB=CHtDxQ@mail.gmail.com>

Explore url module and you need urlretrieve()

saludos,
desde un m?vil.
El nov 12, 2016 12:23 p.m., "Veek M" <vek.m1234 at gmail.com> escribi?:

> Steve D'Aprano wrote:
>
> > On Sat, 12 Nov 2016 11:07 pm, Veek M wrote:
> >
> >> 121sukha wrote:
> >>
> >>> I am new to python and I want to use web scraping to download songs
> >>> from website. how do I write code to check if the website has
> >>> uploaded a new song and have that song automatically be downloaded
> >>> onto my computer. I know how to use the requests.get() module but i
> >>> am more interested in knowing how to download and save every new
> >>> song that the website uploads on the site. I would extremely
> >>> appreciate the help from anyone. Thanks!
> >>
> >> What you're doing is illegal
> >
> > Really? Everywhere in the world? And you're sure of this because
> > you're a lawyer?
>
> 'probably illegal' (i did mentally assume he's from India - the nick is
> very indianish and since I am Indian and noticed many badly formatted
> posts I felt obliged to help out)
>
> > Even if the website uses public domain or freely licenced songs?
> >
> > Even if the downloads count as fair use, time-shifting or
> > format-shifting?
>
> ah true
>
> >> but Youtube/websites have a lot of stuff up
> >> that's easily accessible - the moral issue seems ambiguous especially
> >> if you use adblocker..
> >
> > Doesn't seem ambiguous to me.
>
> > Or are you one of those people who think that you are a thief for
> > leaving the room when ads are playing on TV?
> >
> > https://yro.slashdot.org/story/02/05/02/0550214/turner-
> ceo-pvr-users-are-thieves
>
> yep as in.. do the networks have a right to tempt me
>
> >
> > Downloading may, or may not, violate the terms of use of the website.
> > But you *literally* cannot watch the video without downloading it: in
> > order for the video to play in your browser, it must be downloaded.
> > That's the end of the story. "Streaming video" is just another way of
> > saying "downloading video, where the video is deleted afterwards".
> >
> > (Actually, streaming video is just another way of saying "lots of
> > pauses, stuttering, dropped frames and a really awful user
> > experience".)
>
> also true, but gramophone records were an inconvenience we put up with
> lacking a better solution.. so an inconvenience strictly speaking
> doesn't imply you can break the law.. however, is it fair to tempt us
> with these delights.. hence my moral confusion..
>
> Anyway.. yes I agree that I have no business telling people off (and
> that was not my intention) it was a succinct warning of the dangers..
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From python at mrabarnett.plus.com  Sat Nov 12 14:00:54 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Sat, 12 Nov 2016 19:00:54 +0000
Subject: Python does not start
In-Reply-To: <5826dd49.0d8d1c0a.3fe4e.32d2@mx.google.com>
References: <CAKgG=gyOOca0Ea_W7GfWUY_7dHyYHWr51bvWwScXn=uj9nLa5w@mail.gmail.com>
 <CAKgG=gw+gsE+jyG_6x7Ts-wRh67CJthB3WKuwzhvtZAE6NXwKw@mail.gmail.com>
 <CACL+1at+o1bcgRnNKpGCbntjq4=-j=qnbLtpXmGyCABfnjckfQ@mail.gmail.com>
 <58263a5b.2636c20a.6c87.4e50@mx.google.com>
 <CACL+1auObE6M_1jGu2Jn9dU259e8ZgZV4Q_+1wVJgJCSdenFeg@mail.gmail.com>
 <5826dd49.0d8d1c0a.3fe4e.32d2@mx.google.com>
Message-ID: <c1af0e99-2dc2-75f8-5e65-332b9d2635e9@mrabarnett.plus.com>

On 2016-11-12 09:13, jelena.tavcar at gmail.com wrote:
> I?m sending you the error message from command prompt
>
> C:\Users\Jelena>py.exe -3.5-32 -m idlelib
> Traceback (most recent call last):
[snip]
> _tkinter.TclError: bad event type or keysym "Alt"
>
It looks like someone else has had the same problem and found a fix:

http://bugs.python.org/issue25662

[snip]



From cs at zip.com.au  Sat Nov 12 18:37:24 2016
From: cs at zip.com.au (Cameron Simpson)
Date: Sun, 13 Nov 2016 10:37:24 +1100
Subject: today's gotcha: @property and AttributeError
Message-ID: <20161112233724.GA64699@cskk.homeip.net>

I've just spent a long time debugging what I imagined was a class inheritance 
issue, but which was actually a lack of deep understanding of @property on my 
part.

TL;DR: I am now using this @prop decorator instead of @property in an 
increasing amount of my code:

  def prop(func):
    ''' The builtin @property decorator lets internal AttributeErrors escape.
        While that can support properties that appear to exist conditionally,
        in practice this is almost never what I want, and it masks deeper errors.
        Hence this wrapper for @property that transmutes internal AttributeErrors
        into RuntimeErrors.
    '''
    def wrapper(*a, **kw):
      try:
        return func(*a, **kw)
      except AttributeError as e:
        raise RuntimeError("inner function %s raised %s" % (func, e))
    return property(wrapper)

I was debugging a class with a __getattr__ in the base class and a property in 
the super class, and wondering why the property was not being found. (This was 
a simplification from a dynamicly constructed class with mixins with the same 
problem.)

The root cause was apiece of code like this:

  @property
  def foo(self):
    return self.bah(...blah...)

where self.bah did not exist. This raised AttributeError naturally enough, and 
when a @property has an AttributeError occur the property appears not to exist 
when resolving "foo". And the codefell through to the __getattr_ method from 
the base class, confusing me greatly, because I was looking for a bad MRO 
situation instead of a bug inside the .foo property.

In a sense this is a particular case of a general issue when using exceptions: 
from outside a function one doesn't inherently know if a raised exception is 
"shallow" - from the function one just called, or "deep" - from the inner 
workings of some function deeper in the call stack.

Most of the time this is fine, but when the raised exception is handled my a 
mechanism like @property - in this case to indicate that such a property does 
not exist, a "deep" exception indicating a bug is handled just like a "shallow" 
exception indicating "no property here thanks".

Now that I know what's going on there are plenty of related anecdote apparent 
to web searches, but none offers an alternative to @property such as the code 
above, so I thought I'd share it.

I appreciate that there may be times when I _want_ to raise AttributeError from 
a property to "conceal" it in particular circumstances, but at present that is 
never what i want, and @prop gives me far more debuggable code.

Cheers,
Cameron Simpson <cs at zip.com.au>


From ben+python at benfinney.id.au  Sat Nov 12 22:26:17 2016
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sun, 13 Nov 2016 14:26:17 +1100
Subject: How to distribute your Python software (was: Promoting your own
 library)
References: <66301f14-ac21-4fb8-8405-0af54f9b8ffa@googlegroups.com>
 <8537ix7elo.fsf@benfinney.id.au>
 <895cf749-0a54-100f-6cc1-92da4aaa6d86@harvee.org>
Message-ID: <85oa1k5acm.fsf_-_@benfinney.id.au>

"Eric S. Johansson" <esj at harvee.org> writes:

> I could also use some of this information. I want to publish a couple of
> things that I found useful. My creations tend to be single file modules
> or commands and what I hope to understand from your guidance is how to
> bundle that single file module or standalone program for publication.

The first thing to be aware of is: Distributing your Python software has
been a terrible mess for years, but is getting better steadily because
of the sustained efforts of many.

As a corollary: You will find a lot of bad advice and confusing
contradictions on the internet about how to distribute your Python
software. Be aware that, more than many other technical topics, you will
find *badly misguided* and *outdated* advice from your web searches.

That only gets better by promoting better information, which itself
requires that information to be collated.


So the second thing to know is: Those efforts have a clearing house of
the current best information, at the Python Packaging Authority
<URL:https://packaging.python.org/distributing/>. Today, that is the
first place to start reading about this topic.

-- 
 \        ?A thing is not necessarily true because a man dies for it.? |
  `\                ?Oscar Wilde, _The Portrait of Mr. W. H._, 1889-07 |
_o__)                                                                  |
Ben Finney



From triccare at gmail.com  Sat Nov 12 23:06:49 2016
From: triccare at gmail.com (triccare triccare)
Date: Sat, 12 Nov 2016 23:06:49 -0500
Subject: Why keys method does not work with MutableMapping?
In-Reply-To: <o0515r$2du$1@dont-email.me>
References: <CADtmbDA9tDaq8BHLhfx+K4e_v6ifSnQ1h2KOzVqJygkF3eN6mQ@mail.gmail.com>
 <mailman.107.1478885231.30068.python-list@python.org>
 <o0515r$2du$1@dont-email.me>
Message-ID: <CADtmbDBAGFKni5PhWyTduZfVNnZ5E_Cv_D3pwmZbOg0XLvNvaw@mail.gmail.com>

Ahhh, and what is being shown is simply the __repr__. Gotcha. Thanks!

On Fri, Nov 11, 2016 at 1:03 PM, Rob Gaddi <
rgaddi at highlandtechnology.invalid> wrote:

> triccare triccare wrote:
>
> > Greetings,
> >
> > Apologies if this has shown up twice; I jumped the gun sending before
> > confirming registration.
> >
> > I have a class that completely implements MutableMapping, meaning that
> all
> > the abstract methods are implemented. However, the keys method no longer
> > returns the keys, but simply a repr of the instance.  Example is below.
> > Same is true for the items method.
> >
> > It would seem that, if all the abstract methods have been implemented,
> the
> > keys and items methods should be able to perform exactly like the native
> > dict versions. There does not seem to be a need to override these
> methods.
> >
> > Thank you for your time.
> > triccare
> >
> > Code:
> >
> > from collections import MutableMapping
> >
> > class MyDict(MutableMapping):
> >
> >     def __init__(self, *args, **kwargs):
> >         self.data = dict(*args, **kwargs)
> >
> >     def __getitem__(self, key):
> >         return self.data[self.__keytransform__(key)]
> >
> >     def __setitem__(self, key, value):
> >         self.data[self.__keytransform__(key)] = value
> >
> >     def __delitem__(self, key):
> >         del self.data[self.__keytransform__(key)]
> >
> >     def __iter__(self):
> >         return iter(self.data)
> >
> >     def __len__(self):
> >         return len(self.data)
> >
> >     def __keytransform__(self, key):
> >         return key
> >
> > md = MyDict({'a': 1, 'b':2})
> > md.keys()
> > ==> KeysView(<keys.MyDict object at 0x105f7f7b8>)
>
> Nope, that's exactly right.  That's the python3 behavior.
>
> >>> d = {'a': 1, 'b':2}
> >>> d.keys()
> dict_keys(['b', 'a'])
>
> Keys returns a dedicated keys object now, not just a list.  That thing
> you got back isn't a repr string; it's the actual object.  If it were a
> string it'd be quoted.
>
> Try list(md.keys()).
>
>
> --
> Rob Gaddi, Highland Technology -- www.highlandtechnology.com
> Email address domain is currently out of order.  See above to fix.
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From kfjwheeler at gmail.com  Sun Nov 13 02:04:18 2016
From: kfjwheeler at gmail.com (kfjwheeler at gmail.com)
Date: Sat, 12 Nov 2016 23:04:18 -0800 (PST)
Subject: Best way to go about embedding python
Message-ID: <b6ac3d58-28db-4539-8234-b6652dc0b8a3@googlegroups.com>


A friend of mine is developing a game engine, and I would like to extend the engine by adding some scripting functionality.  However, there's a lot of information out there 


From kfjwheeler at gmail.com  Sun Nov 13 02:10:50 2016
From: kfjwheeler at gmail.com (kfjwheeler at gmail.com)
Date: Sat, 12 Nov 2016 23:10:50 -0800 (PST)
Subject: Best way to go about embedding python
Message-ID: <16cbd5d3-e2df-4635-bd4e-5772cb079c6f@googlegroups.com>


(Sorry about the last post, my finger slipped before I was finished, and I can't delete it for some reason.)

A friend of mine is developing a game engine, and I would like to extend the engine by adding some scripting functionality.  However, there's a lot of information out there, and I have a lot of questions before I get started.

1.  What's the best implementation to use?
        - The engine itself is written in pure C, and the compiler being used is clang.  Eventually, we'd like it to target multiple platforms.

2.  How would I go about including python scripts to run with the engine.  Would the .py files from the standard libraries be necessary, or is anything built in?

3.  How could I go about adding a repl to the engine application?  Preferably, I'd be able to connect to it with sublimerepl.

4.  Would it be possible to attach any python debuggers to the application and step through python code?


I realize that, in some cases, I may not even be asking the right questions, but any information that could point me in the right direction would be greatly appreciated.

Thanks


From ben+python at benfinney.id.au  Sun Nov 13 03:20:41 2016
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sun, 13 Nov 2016 19:20:41 +1100
Subject: Best way to go about embedding python
References: <16cbd5d3-e2df-4635-bd4e-5772cb079c6f@googlegroups.com>
Message-ID: <85k2c76bae.fsf@benfinney.id.au>

kfjwheeler at gmail.com writes:

> A friend of mine is developing a game engine, and I would like to
>  extend the engine by adding some scripting functionality.

I haven't done it myself. But the Python Wiki has some pages that will
help you, I think:

* <URL:https://wiki.python.org/moin/AppsWithPythonScripting>
* <URL:https://wiki.python.org/moin/IntegratingPythonWithOtherLanguages>

-- 
 \       ?The internet's completely over.? Anyway, all these computers |
  `\    and digital gadgets are no good. They just fill your head with |
_o__)     numbers and that can't be good for you.? ?Prince, 2010-07-05 |
Ben Finney



From rosuav at gmail.com  Sun Nov 13 08:23:39 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 14 Nov 2016 00:23:39 +1100
Subject: Best way to go about embedding python
In-Reply-To: <16cbd5d3-e2df-4635-bd4e-5772cb079c6f@googlegroups.com>
References: <16cbd5d3-e2df-4635-bd4e-5772cb079c6f@googlegroups.com>
Message-ID: <CAPTjJmpsYiBWYjMi2QKQmkTYG4Ag--m_f9n2BmWFFVWSyiQa+Q@mail.gmail.com>

On Sun, Nov 13, 2016 at 6:10 PM,  <kfjwheeler at gmail.com> wrote:
>
> (Sorry about the last post, my finger slipped before I was finished, and I can't delete it for some reason.)

(That's because this is a mailing list and newsgroup gatewayed to each
other. Once sent, a message cannot be edited or deleted.)

> A friend of mine is developing a game engine, and I would like to extend the engine by adding some scripting functionality.  However, there's a lot of information out there, and I have a lot of questions before I get started.
>

The very first question to ask is: Are you intending to let untrusted
or less trusted people write the scripts? If so, *do not* use Python.
Embedding Python can be done to make your work easier, but it isn't
sandboxed.

> 1.  What's the best implementation to use?
>         - The engine itself is written in pure C, and the compiler being used is clang.  Eventually, we'd like it to target multiple platforms.
>

Sounds like the standard CPython interpreter will suit you just fine.

> 2.  How would I go about including python scripts to run with the engine.  Would the .py files from the standard libraries be necessary, or is anything built in?
>

You'll probably want to link against libpython, which (AIUI) will
connect you to an installed Python, complete with the standard
library. Python without its stdlib is a disappointingly featureless
language :)

> 3.  How could I go about adding a repl to the engine application?  Preferably, I'd be able to connect to it with sublimerepl.
>

Hmm, that depends a bit on the way your app works. Most likely, a
classic REPL won't work, as the console won't be directly available.
You'll probably end up with an internal console of some sort. You
should be able to either build your own REPL on top of that, or tie in
with something from Idle's code.

> 4.  Would it be possible to attach any python debuggers to the application and step through python code?
>

Again, this depends on how your app is architected, but it should be possible.

Embedding Python isn't too hard. The best place to start is here:

https://docs.python.org/3/extending/index.html

You'll find it's easy enough to get a simple "Hello, world" going, and
after that, it's a matter of figuring out what you need.

Have fun with it!

ChrisA


From wants at no.mail.local  Sun Nov 13 09:12:01 2016
From: wants at no.mail.local (andy)
Date: Sun, 13 Nov 2016 15:12:01 +0100
Subject: how to print variable few time?
References: <7b3d77cb-c42d-48bb-876f-ef36d9dc27f4@googlegroups.com>
Message-ID: <d5251$582874b1$54817df3$25998@news1.surfino.com>

Sat, 12 Nov 2016 04:58:20 -0800 wrote guy asor:

> hello!
> 
> this is my code:
> 
> word=raw_input()
> print word*3
> 
> 
> with this code im getting - wordwordword.
> what changes i need to make to get - word word word - instead?
> 
> thanks

using python3.x:

word=input()
print((word+' ')*2, end='')
print(word)

 ..but D'Apranos ' '.join(..) is far more elegant.


From wants at no.mail.local  Sun Nov 13 10:03:06 2016
From: wants at no.mail.local (andy)
Date: Sun, 13 Nov 2016 16:03:06 +0100
Subject: how to print variable few time?
References: <7b3d77cb-c42d-48bb-876f-ef36d9dc27f4@googlegroups.com>
 <d5251$582874b1$54817df3$25998@news1.surfino.com>
Message-ID: <b76e8$582880aa$54817df3$25998@news1.surfino.com>

Sun, 13 Nov 2016 15:12:01 +0100 wrote andy:

> word=input()
> print((word+' ')*2, end='')
> print(word)
> 
>  ..but D'Apranos ' '.join(..) is far more elegant.

don't know whether I really would use this form, but it works in python3:

word = ' '.join([input()]*3)
print(word)


From jussi.piitulainen at helsinki.fi  Sun Nov 13 10:27:23 2016
From: jussi.piitulainen at helsinki.fi (Jussi Piitulainen)
Date: Sun, 13 Nov 2016 17:27:23 +0200
Subject: how to print variable few time?
References: <7b3d77cb-c42d-48bb-876f-ef36d9dc27f4@googlegroups.com>
 <d5251$582874b1$54817df3$25998@news1.surfino.com>
Message-ID: <lf5polzflic.fsf@ling.helsinki.fi>

andy writes:

> Sat, 12 Nov 2016 04:58:20 -0800 wrote guy asor:
>
>> hello!
>> 
>> this is my code:
>> 
>> word=raw_input()
>> print word*3
>> 
>> 
>> with this code im getting - wordwordword.
>> what changes i need to make to get - word word word - instead?
>> 
>> thanks
>
> using python3.x:
>
> word=input()
> print((word+' ')*2, end='')
> print(word)

print(*[word]*3)


From antoon.pardon at rece.vub.ac.be  Sun Nov 13 15:02:33 2016
From: antoon.pardon at rece.vub.ac.be (Antoon Pardon)
Date: Sun, 13 Nov 2016 21:02:33 +0100
Subject: if iter(iterator) is iterator
Message-ID: <f803560e-6af3-735e-735c-666437e1d5d0@rece.vub.ac.be>


Some time ago I read a text or saw a video on iterators and one thing
I remember from it, is that you should do something like the following
when working with iterators, to avoid some pitt falls.

def bar(iterator):
    if iter(iterator) is iterator:
        ...

However I can't relocate it and can't remember the what and how anymore.
Does anyone know what this is about?

-- 
Antoon.


From menta231 at gmail.com  Sun Nov 13 15:09:40 2016
From: menta231 at gmail.com (menta231 at gmail.com)
Date: Sun, 13 Nov 2016 12:09:40 -0800 (PST)
Subject: Halp in if
Message-ID: <166a69a4-058f-49ce-be18-8b2245416b6f@googlegroups.com>

Hii there
I am preety New at python,  and i need your help. I want to make a script that if somthing happens, it stop, and if it doesnt, it couninues. I dont finding a smart way. Here is what i wrote. I want to do it without exits and sleeps. 


import time
print ("hello there ")
x = raw_input ("what is you name ? ")
print ("hello ") + x + (" lets play a game ")
y = raw_input ("chooe a number ")
iny = raw_input ("what is the power of this number ?")
z = int(y) * int(y)
if int(iny) == int(y) * int(y) :
 print ("I see you are smart ")
else: 
 print ("Such An Idiot! answer is ") + str(z)
 time.sleep(3)
 exit()
if int(iny) == int(y) * int(y) :
 print ("think touy are so smart")
lll=raw_input("do you want to cotinue?")
if lll== str("yes"):
 print ("good")
else:
 print ("bye bye")
 time.sleep(2)
 exit()
lmm=raw_input("what is the thirs power of the same number")
if lmm == str(int(y)* int(y)*int(y)):
 print ("you are a pro")
else :
 print ("I'm dissapointed... answear is ") + str(int(y) * int(y)*int(y))
 time.sleep (5)
 exit()
time.sleep(121)


From rosuav at gmail.com  Sun Nov 13 15:19:14 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 14 Nov 2016 07:19:14 +1100
Subject: if iter(iterator) is iterator
In-Reply-To: <f803560e-6af3-735e-735c-666437e1d5d0@rece.vub.ac.be>
References: <f803560e-6af3-735e-735c-666437e1d5d0@rece.vub.ac.be>
Message-ID: <CAPTjJmr_y55-pXZKGmRtkXFLSYGFhpo07qVpd5pbfvxn9iB2pA@mail.gmail.com>

On Mon, Nov 14, 2016 at 7:02 AM, Antoon Pardon
<antoon.pardon at rece.vub.ac.be> wrote:
> Some time ago I read a text or saw a video on iterators and one thing
> I remember from it, is that you should do something like the following
> when working with iterators, to avoid some pitt falls.
>
> def bar(iterator):
>     if iter(iterator) is iterator:
>         ...
>
> However I can't relocate it and can't remember the what and how anymore.
> Does anyone know what this is about?

That's how you distinguish an iterator from other iterables. An
iterable is something you can call iter() on; an iterator is required
to specifically return _itself_.

What you do with that information, though, depends on the function. In
a lot of cases, you shouldn't care - if you need an iterator, just
"iterator = iter(iterator)" at the top and move on.

ChrisA


From greg.ewing at canterbury.ac.nz  Sun Nov 13 16:57:01 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Mon, 14 Nov 2016 10:57:01 +1300
Subject: if iter(iterator) is iterator
In-Reply-To: <mailman.129.1479067429.30068.python-list@python.org>
References: <f803560e-6af3-735e-735c-666437e1d5d0@rece.vub.ac.be>
 <mailman.129.1479067429.30068.python-list@python.org>
Message-ID: <e8s5tfFkqjU1@mid.individual.net>

Antoon Pardon wrote:

> def bar(iterator):
>     if iter(iterator) is iterator:
>         ...

That's a way of finding out whether you can safely iterate
over something more than once. If the object is already an
iterator, applying iter() to it will return the same
object, and iterating over it will consume it.

You would only care if you're intending to iterate over
it more than once, and you're not sure what kind of object
you have.

-- 
Greg


From python at lucidity.plus.com  Sun Nov 13 17:00:34 2016
From: python at lucidity.plus.com (Erik)
Date: Sun, 13 Nov 2016 22:00:34 +0000
Subject: Halp in if
In-Reply-To: <166a69a4-058f-49ce-be18-8b2245416b6f@googlegroups.com>
References: <166a69a4-058f-49ce-be18-8b2245416b6f@googlegroups.com>
Message-ID: <d5a7bc09-d924-4419-398a-284675bbead4@lucidity.plus.com>

Hi,

On 13/11/16 20:09, menta231 at gmail.com wrote:
> HI want to make a script that if somthing happens, it stop, and if it doesnt, it couninues. I dont finding a smart way.

Read the documentation for the "while", "break" and "continue" keywords.

Regards, E.



From rosuav at gmail.com  Sun Nov 13 17:04:50 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 14 Nov 2016 09:04:50 +1100
Subject: if iter(iterator) is iterator
In-Reply-To: <e8s5tfFkqjU1@mid.individual.net>
References: <f803560e-6af3-735e-735c-666437e1d5d0@rece.vub.ac.be>
 <mailman.129.1479067429.30068.python-list@python.org>
 <e8s5tfFkqjU1@mid.individual.net>
Message-ID: <CAPTjJmr5OhPHTd5=Gw32aMyOvdJkVZxqaPmkecG1u+eZDFgCQQ@mail.gmail.com>

On Mon, Nov 14, 2016 at 8:57 AM, Gregory Ewing
<greg.ewing at canterbury.ac.nz> wrote:
> Antoon Pardon wrote:
>
>> def bar(iterator):
>>     if iter(iterator) is iterator:
>>         ...
>
>
> That's a way of finding out whether you can safely iterate
> over something more than once. If the object is already an
> iterator, applying iter() to it will return the same
> object, and iterating over it will consume it.

Not strictly true. There's no guarantee that something that yields new
iterators is actually reiterable. And even if it is, there's no
guarantee that it'll yield the same values.

ChrisA


From priisdk at gmail.com  Sun Nov 13 17:56:01 2016
From: priisdk at gmail.com (Poul Riis)
Date: Sun, 13 Nov 2016 14:56:01 -0800 (PST)
Subject: Making stl files with python for 3d printing
Message-ID: <53f713f9-bf6e-4b58-869a-0b18f3c6bc2a@googlegroups.com>

Below you can find my python code to make a vase for 3D-printing.
The vase looks nice on my screen so the vertices and the faces seem to be perfect.
However, when sending the .stl file produced to a 3D-printer the vase comes out as a filled solid - no room for water and flowers!
I have tried to slice the vase the the program "Cura". It seems that the slices are full disks, not rings as intended, at some values of z but far from all.
I don't expect anybody to read the full program (I'm a very poor programmer) but my hope is that someone can give a hint as to how I can improve the writing of the .stl file.

Poul Riis

#!/usr/bin/env python
#coding:utf-8
# Purpose: Export 3D objects, build of faces with 3 or 4 vertices, as ASCII or Binary STL file.
# License: MIT License

import struct
from tvtk.api import tvtk
from mayavi import mlab
from math import *
from sympy import *



ASCII_FACET = """facet normal 0 0 0
outer loop
vertex {face[0][0]:.4f} {face[0][1]:.4f} {face[0][2]:.4f}
vertex {face[1][0]:.4f} {face[1][1]:.4f} {face[1][2]:.4f}
vertex {face[2][0]:.4f} {face[2][1]:.4f} {face[2][2]:.4f}
endloop
endfacet
"""

BINARY_HEADER ="80sI"
BINARY_FACET = "12fH"

class ASCII_STL_Writer:
    """ Export 3D objects build of 3 or 4 vertices as ASCII STL file.
    """
    def __init__(self, stream):
        self.fp = stream
        self._write_header()

    def _write_header(self):
        self.fp.write("solid python\n")

    def close(self):
        self.fp.write("endsolid python\n")

    def _write(self, face):
        self.fp.write(ASCII_FACET.format(face=face))

    def _split(self, face):
        p1, p2, p3, p4 = face
        return (p1, p2, p3), (p3, p4, p1)

    def add_face(self, face):
        """ Add one face with 3 or 4 vertices. """
        if len(face) == 4:
            face1, face2 = self._split(face)
            self._write(face1)
            self._write(face2)
        elif len(face) == 3:
            self._write(face)
        else:
            raise ValueError('only 3 or 4 vertices for each face')

    def add_faces(self, faces):
        """ Add many faces. """
        for face in faces:
            self.add_face(face)

class Binary_STL_Writer(ASCII_STL_Writer):
    """ Export 3D objects build of 3 or 4 vertices as binary STL file.
    """
    def __init__(self, stream):
        self.counter = 0
        super(Binary_STL_Writer, self).__init__(stream)

    def close(self):
        self._write_header()

    def _write_header(self):
        self.fp.seek(0)
        self.fp.write(struct.pack(BINARY_HEADER, b'Python Binary STL Writer', self.counter))

    def _write(self, face):
        self.counter += 1
        data = [
            0., 0., 0.,
            face[0][0], face[0][1], face[0][2],
            face[1][0], face[1][1], face[1][2],
            face[2][0], face[2][1], face[2][2],
            0
        ]
        self.fp.write(struct.pack(BINARY_FACET, *data))


def example(fn):
    def get_cube():
        p=[]
        nxy=24
        nz=21#Must be odd
        unit=10
        h=2*unit
        dh=h/nz
        zbottom=-2*dh
        thickness=0.3*unit
        
        z=Symbol('z')
        f=1*(1+sin((z/h)**2*1*pi)/8)*exp(z/h/2)*unit#1*
        #f=(1+z/h)*unit
        flambdified = lambdify(z, f)
        fdiff=f.diff(z)
        fdifflambdified = lambdify(z, fdiff)
        
        rinner0=10
        rinner=rinner0
        router0=rinner0+thickness
        router=router0
        deltar=1
        deltaphi=2*pi/nxy
        deltaphih=deltaphi/2
        nxyz=nxy*nz
        npts=0
        
        #Inner:
        for j in range(0,nz+1):
            zact=j*dh
            ract=flambdified(zact)
            phiact=j%2*(-deltaphih)
            for i in range(0,nxy):
                p.append((ract*cos(phiact),ract*sin(phiact),zact))
                phiact=phiact+deltaphi
                npts=npts+1
        npts1=npts
        

        #Middle of upper rim:
        phiact=0
        ract=flambdified(zact)
        a=fdifflambdified(zact)
        b=sqrt(1+a*a)
        k=thickness/b/2
        for i in range(0,nxy):
            cosphi=cos(phiact)
            sinphi=sin(phiact)        
            dx=k*cosphi
            dy=k*sinphi
            dz=-k*a
            p.append((ract*cosphi+dx,ract*sinphi+dy,zact+dz))
            phiact=phiact+deltaphi
            npts1=npts1+1
        
        #Outer:
        for j in range(-1,nz+2):
            zact=(nz-j-1)*dh
            ract=flambdified(zact)
            a=fdifflambdified(zact)
            b=sqrt(1+a*a)
            k=thickness/b            
            phiact=(j-0)%2*(-deltaphih)
            for i in range(0,nxy):
                cosphi=cos(phiact)
                sinphi=sin(phiact)        
                dx=k*cosphi
                dy=k*sinphi
                dz=-k*a
                p.append((ract*cosphi+dx,ract*sinphi+dy,zact+dz))
                phiact=phiact+deltaphi
                npts1=npts1+1

       
        #Center of bottom:
        p.append((0,0,-thickness))

        parray=[]
        
        #Inner:
        for j in range(0,nz*2+4,2):
            for i in range(0,nxy):
                i0=i%nxy+j*nxy
                i1=(i+1)%nxy+j*nxy
                i2=(i+1)%nxy+nxy+j*nxy
                i3=i%nxy+nxy+j*nxy
                parray.append([p[i0],p[i1],p[i2],p[i3]])
            
            for i in range(0,nxy):
                i0=(i+1)%nxy+nxy+j*nxy
                i1=i%nxy+nxy+j*nxy
                i2=i%nxy+nxy+nxy+j*nxy
                i3=(i+1)%nxy+nxy+nxy+j*nxy               
                parray.append([p[i0],p[i1],p[i2],p[i3]])
            
        #Inner bottom:   
        for i in range(0,nxy):
            parray.append([(0,0,0),p[i%nxy],p[(i+1)%nxy]])

        #Outer bottom:        
        for i in range(0,nxy):
            parray.append([(0,0,zbottom),p[npts1-i%nxy-1],p[npts1-(i+1)%nxy-1]])
        return parray

    filename=fn+'.stl'
    with open(filename, 'wb') as fp:
        writer = Binary_STL_Writer(fp)
        writer.add_faces(get_cube())
        writer.close()
    print(filename,' saved.')
    

    # Finally the plotting:
    print('Finally the plotting....')
    stlr = tvtk.STLReader()
    stlr.file_name = file_name=filename
    stlr.update()
    stld = stlr.output
    fig = mlab.figure(bgcolor=(1, 1, 1), fgcolor=(0, 0, 0))
    surf = mlab.pipeline.surface(stld, opacity=1, color=(0.3, 0.8, 0.7))
    edge = mlab.pipeline.extract_edges(surf)
    edge_surf = mlab.pipeline.surface(edge, opacity=.1, color=(1,0,0))


        

if __name__ == '__main__':
    example('vase_simple')






From steve+python at pearwood.info  Sun Nov 13 19:40:18 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Mon, 14 Nov 2016 11:40:18 +1100
Subject: if iter(iterator) is iterator
References: <f803560e-6af3-735e-735c-666437e1d5d0@rece.vub.ac.be>
 <mailman.129.1479067429.30068.python-list@python.org>
Message-ID: <582907f5$0$1607$c3e8da3$5496439d@news.astraweb.com>

On Mon, 14 Nov 2016 07:02 am, Antoon Pardon wrote:

> 
> Some time ago I read a text or saw a video on iterators and one thing
> I remember from it, is that you should do something like the following
> when working with iterators, to avoid some pitt falls.
> 
> def bar(iterator):
>     if iter(iterator) is iterator:
>         ...
> 
> However I can't relocate it and can't remember the what and how anymore.
> Does anyone know what this is about?

Probably something like this:


def function(iterable):
    if iter(iterable) is iterable:
        print("iterable is already an iterator")
        assert hasattr(iterable, '__next__')  # 'next' in Python 2
    else:
        print("iterable is not an iterator")
        print("it may be a list, dict, string, etc.")


`iter(thing) is thing` is an easy way to check whether thing is an iterator.

I'm surprised that the inspect module doesn't appear to have isiterable and
isiterator functions. Here's my first attempt at both:


def isiterable(obj):
    """Return True if obj is an iterable, and False otherwise.

    Iterable objects can be iterated over, and they provide either
    an __iter__ method or a __getitem__ method.

    Iteration over an object tries calling the object's __iter__ 
    method (if any), then repeatedly calls __next__ on the result
    until a StopIteration exception is raised (the Iterator
    Protocol). Otherwise, it tries calling __getitem__ with 
    arguments 0, 1, 2, 3 ... until an IndexError exception is
    raised (the Sequence Protocol).
    
    """
    T = type(obj)
    return hasattr(T, '__iter__') or hasattr(T, '__getitem__')


def isiterator(obj):
    """Return True if obj is an iterator, and False otherwise.

    Iterators can be iterated over, and they provide the following
    methods:

    - __iter__ which returns itself
    - __next__

    Iteration over an iterator repeatedly calls __next__ until a
    StopIteration exception is raised.
    """
    T = type(obj)
    if hasattr(T, '__iter__') and hasattr(T, '__next__'):
        return iter(obj) is obj
    return False






-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Sun Nov 13 19:40:46 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Mon, 14 Nov 2016 11:40:46 +1100
Subject: Best way to go about embedding python
References: <16cbd5d3-e2df-4635-bd4e-5772cb079c6f@googlegroups.com>
 <CAPTjJmpsYiBWYjMi2QKQmkTYG4Ag--m_f9n2BmWFFVWSyiQa+Q@mail.gmail.com>
 <mailman.127.1479043421.30068.python-list@python.org>
Message-ID: <5829080f$0$1607$c3e8da3$5496439d@news.astraweb.com>

On Mon, 14 Nov 2016 12:23 am, Chris Angelico wrote:

> Python without its stdlib is a disappointingly featureless
> language :)


I don't think that's true. You can do a lot in Python without any imports in
your code:

- basic string processing
- BigNum (int) and float arithmetic
- lists and tuples
- dicts
- lazy processing of iterators
- custom functions and classes
- functional style zip, map and filter

and more.

Some parts of the standard library are required for built-in functions, e.g.
the io module is needed to open files. But I think you could probably
reduce the standard library by, oh, 90% and still have a decent language
for game scripting.


-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From rosuav at gmail.com  Sun Nov 13 19:59:22 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 14 Nov 2016 11:59:22 +1100
Subject: if iter(iterator) is iterator
In-Reply-To: <582907f5$0$1607$c3e8da3$5496439d@news.astraweb.com>
References: <f803560e-6af3-735e-735c-666437e1d5d0@rece.vub.ac.be>
 <mailman.129.1479067429.30068.python-list@python.org>
 <582907f5$0$1607$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPTjJmoOCnyRWg5FWUKrsWaqi5JFFXaA_Crb9NzxP7f7R=6WAg@mail.gmail.com>

On Mon, Nov 14, 2016 at 11:40 AM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> def isiterable(obj):
>     """Return True if obj is an iterable, and False otherwise.
>
>     Iterable objects can be iterated over, and they provide either
>     an __iter__ method or a __getitem__ method.
>
>     Iteration over an object tries calling the object's __iter__
>     method (if any), then repeatedly calls __next__ on the result
>     until a StopIteration exception is raised (the Iterator
>     Protocol). Otherwise, it tries calling __getitem__ with
>     arguments 0, 1, 2, 3 ... until an IndexError exception is
>     raised (the Sequence Protocol).
>
>     """
>     T = type(obj)
>     return hasattr(T, '__iter__') or hasattr(T, '__getitem__')

Any particular reason to write it that way, rather than:

def isiterable(obj):
    try:
        iter(obj)
        return True
    except TypeError:
        return False

?

ChrisA


From rosuav at gmail.com  Sun Nov 13 20:03:10 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 14 Nov 2016 12:03:10 +1100
Subject: Best way to go about embedding python
In-Reply-To: <5829080f$0$1607$c3e8da3$5496439d@news.astraweb.com>
References: <16cbd5d3-e2df-4635-bd4e-5772cb079c6f@googlegroups.com>
 <CAPTjJmpsYiBWYjMi2QKQmkTYG4Ag--m_f9n2BmWFFVWSyiQa+Q@mail.gmail.com>
 <mailman.127.1479043421.30068.python-list@python.org>
 <5829080f$0$1607$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPTjJmoFQoV+zsL5C8DzvXS98A89XLtbWhu44Prgt1ro+jAvnQ@mail.gmail.com>

On Mon, Nov 14, 2016 at 11:40 AM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> On Mon, 14 Nov 2016 12:23 am, Chris Angelico wrote:
>
>> Python without its stdlib is a disappointingly featureless
>> language :)
>
>
> I don't think that's true. You can do a lot in Python without any imports in
> your code:
>
> - basic string processing
> - BigNum (int) and float arithmetic
> - lists and tuples
> - dicts
> - lazy processing of iterators
> - custom functions and classes
> - functional style zip, map and filter
>
> and more.
>
> Some parts of the standard library are required for built-in functions, e.g.
> the io module is needed to open files. But I think you could probably
> reduce the standard library by, oh, 90% and still have a decent language
> for game scripting.

Perhaps what would be more accurate is that Python without its stdlib
is as featureless as (say) Lua, but still without being safe to embed.
Yes, it's technically usable, but you really don't have a lot. And
remember, some of the stdlib is what we call builtins; if you embed
Python and engage "restricted mode", it's by emptying out the
builtins. You're pretty much down to three things:

1) Stuff that you could pass to ast.literal_eval
2) Control flow statements
3) Hacks that get you around the limitations, thus proving that
Python-without-stdlib is really really hard to actually test anything
with.

ChrisA


From torriem at gmail.com  Sun Nov 13 20:15:43 2016
From: torriem at gmail.com (Michael Torrie)
Date: Sun, 13 Nov 2016 18:15:43 -0700
Subject: Best way to go about embedding python
In-Reply-To: <16cbd5d3-e2df-4635-bd4e-5772cb079c6f@googlegroups.com>
References: <16cbd5d3-e2df-4635-bd4e-5772cb079c6f@googlegroups.com>
Message-ID: <713d8eaf-030c-3874-19ee-7771545b1dbe@gmail.com>

On 11/13/2016 12:10 AM, kfjwheeler at gmail.com wrote:
> 2.  How would I go about including python scripts to run with the
> engine.  Would the .py files from the standard libraries be
> necessary, or is anything built in?

It's usually just a matter of asking the python instance to load and run
the script for you.  You'll need to be familiar with the libpython C
API.  This document is a good starting point:
https://docs.python.org/3/extending/extending.html

One of the interesting things about libpython and the API is that
there's really little difference between extending Python with C and
embedding Python in C.  The principles are the same.

The hardest part about integrating Python is that you have to consider
what kinds of primitives from your C program you want to expose to
Python and how you intend to expose them.  You'll have to use the C api
to wrap your native structures and functions so that they can be
interacted with from Python.  And such bindings may need to be two-way.
For example in Python code you may want to instantiate some new instance
of game object, and you'll need to make sure that the wrapper code will
create the appropriate in-game object. This aspect is probably going to
be the most time-consuming and hardest.

I'm glad to hear of someone wanting to embed Python.  Seems like so many
projects are using Javascript (yuck).  The Epiphany browser, for example
used to use Python for creating extensions but years ago they tossed it
out in favor of javascript which was a step backwards I thought.  Of
course Lua is probably the most popular embedded scripting language,
particularly with gaming engines.


From nathan.ernst at gmail.com  Sun Nov 13 20:38:00 2016
From: nathan.ernst at gmail.com (Nathan Ernst)
Date: Sun, 13 Nov 2016 19:38:00 -0600
Subject: Best way to go about embedding python
In-Reply-To: <CAPTjJmoFQoV+zsL5C8DzvXS98A89XLtbWhu44Prgt1ro+jAvnQ@mail.gmail.com>
References: <16cbd5d3-e2df-4635-bd4e-5772cb079c6f@googlegroups.com>
 <CAPTjJmpsYiBWYjMi2QKQmkTYG4Ag--m_f9n2BmWFFVWSyiQa+Q@mail.gmail.com>
 <mailman.127.1479043421.30068.python-list@python.org>
 <5829080f$0$1607$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmoFQoV+zsL5C8DzvXS98A89XLtbWhu44Prgt1ro+jAvnQ@mail.gmail.com>
Message-ID: <CAPjBxg8GNqkLPqUSjomTEofk=RAff_kMKxYi3-vA+q2b0Xu9Vw@mail.gmail.com>

In regards to performance of Lua vs Python, I don't have enough (near zero
experience) with Lua to comment there.

But in regards to embedding in a game, the only experience I have w/ Python
being embedded is while working on modding Civilization IV. What I saw
there just made me nauseous.

The reasoning for moving to Lua in Civilization V was performance & memory
usage - but they completely screwed up exposing the C++ internals to Python
in Civ IV. They copied object instances & containers to the Python
embedded, then marshalled them back to the C++ code when the Python code
was finished. It was horribly inefficient.

I don't have much experience hosting Python inside of a C/C++ app, but I've
10 years of experience consuming C++ APIs from Python, mostly wrapped using
Boost Python. I always did my best to minimize data marshalling. The Python
objects exposed the C++ objects' interface, directly while holding a
pointer (usually shared) to underlying C++ objects and only marshalling on
method/property calls where needed (i.e. w/ strings, dates, datetimes).
Containers were wrapped to expose a Python interface to an underlying C++
container w/ no copying.

TLDR; Civilization IV shows an antipattern about how to embed Python in a
native app and expecting anything resembling good performance. Civ IV used
Boost Python, but they did it in an extremely poor, suboptimal fashion.
Boost Python itself can be very performant, and I love the library. Boost
Python, however does suffer from a learning curve that is akin to
attempting to scale a 1000ft cliff with no rope or tools or even shoes. Due
to the heavy use of template metaprogramming in C++, compiler errors if
you're even just slightly off can be extremely hard to decipher. I've not
used it lately or on modern compilers, so I don't know if it's gotten
better, but tools like pretty_make.py (can find on github) can help filter
out the compiler spewage from the real compiler error.

PS. I don't think that Python couldn't/shouldn't be used for scripting a
game, I'm just offering up Civ IV as an example of how it can be done
horribly wrong.

Regards,
Nate

On Sun, Nov 13, 2016 at 7:03 PM, Chris Angelico <rosuav at gmail.com> wrote:

> On Mon, Nov 14, 2016 at 11:40 AM, Steve D'Aprano
> <steve+python at pearwood.info> wrote:
> > On Mon, 14 Nov 2016 12:23 am, Chris Angelico wrote:
> >
> >> Python without its stdlib is a disappointingly featureless
> >> language :)
> >
> >
> > I don't think that's true. You can do a lot in Python without any
> imports in
> > your code:
> >
> > - basic string processing
> > - BigNum (int) and float arithmetic
> > - lists and tuples
> > - dicts
> > - lazy processing of iterators
> > - custom functions and classes
> > - functional style zip, map and filter
> >
> > and more.
> >
> > Some parts of the standard library are required for built-in functions,
> e.g.
> > the io module is needed to open files. But I think you could probably
> > reduce the standard library by, oh, 90% and still have a decent language
> > for game scripting.
>
> Perhaps what would be more accurate is that Python without its stdlib
> is as featureless as (say) Lua, but still without being safe to embed.
> Yes, it's technically usable, but you really don't have a lot. And
> remember, some of the stdlib is what we call builtins; if you embed
> Python and engage "restricted mode", it's by emptying out the
> builtins. You're pretty much down to three things:
>
> 1) Stuff that you could pass to ast.literal_eval
> 2) Control flow statements
> 3) Hacks that get you around the limitations, thus proving that
> Python-without-stdlib is really really hard to actually test anything
> with.
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From jfong at ms4.hinet.net  Sun Nov 13 21:39:03 2016
From: jfong at ms4.hinet.net (jfong at ms4.hinet.net)
Date: Sun, 13 Nov 2016 18:39:03 -0800 (PST)
Subject: help on "from deen import *" vs. "import deen"
Message-ID: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>

Running the following codes (deen.py) under Win32 python 3.4.4 terminal:

tbli = [0x66, 0x27, 0xD0]
tblm = [0 for x in range(3)]
def gpa(data):
    td = data ^ tblm[2]
    return td

I can get a correct answer this way:
>>> import deen
>>> deen.tblm = deen.tbli
>>> deen.gpa(0x7d)
173  # 0xad (= 0x7d ^ 0xd0) is decimal 173

But I get a wrong answer this way:
>>> from deen import *
>>> tblm = tbli
>>> gpa(0x7d)
125  # it's 0x7d, the tblm[2] is 0

Why? why! why:-(

--Jach



From kfjwheeler at gmail.com  Sun Nov 13 22:40:47 2016
From: kfjwheeler at gmail.com (Kaylen Wheeler)
Date: Sun, 13 Nov 2016 19:40:47 -0800 (PST)
Subject: Best way to go about embedding python
In-Reply-To: <mailman.137.1479087490.30068.python-list@python.org>
References: <16cbd5d3-e2df-4635-bd4e-5772cb079c6f@googlegroups.com>
 <CAPTjJmpsYiBWYjMi2QKQmkTYG4Ag--m_f9n2BmWFFVWSyiQa+Q@mail.gmail.com>
 <mailman.127.1479043421.30068.python-list@python.org>
 <5829080f$0$1607$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmoFQoV+zsL5C8DzvXS98A89XLtbWhu44Prgt1ro+jAvnQ@mail.gmail.com>
 <CAPjBxg8GNqkLPqUSjomTEofk=RAff_kMKxYi3-vA+q2b0Xu9Vw@mail.gmail.com>
 <mailman.137.1479087490.30068.python-list@python.org>
Message-ID: <c8ac8ef8-a323-4a26-8ed4-ae3458faa669@googlegroups.com>

I wonder if Lua would be a better option.  Does it suffer from the same sandboxing issues that python does?

On Sunday, 13 November 2016 17:38:23 UTC-8, Nathan Ernst  wrote:
> In regards to performance of Lua vs Python, I don't have enough (near zero
> experience) with Lua to comment there.
> 
> But in regards to embedding in a game, the only experience I have w/ Python
> being embedded is while working on modding Civilization IV. What I saw
> there just made me nauseous.
> 
> The reasoning for moving to Lua in Civilization V was performance & memory
> usage - but they completely screwed up exposing the C++ internals to Python
> in Civ IV. They copied object instances & containers to the Python
> embedded, then marshalled them back to the C++ code when the Python code
> was finished. It was horribly inefficient.
> 
> I don't have much experience hosting Python inside of a C/C++ app, but I've
> 10 years of experience consuming C++ APIs from Python, mostly wrapped using
> Boost Python. I always did my best to minimize data marshalling. The Python
> objects exposed the C++ objects' interface, directly while holding a
> pointer (usually shared) to underlying C++ objects and only marshalling on
> method/property calls where needed (i.e. w/ strings, dates, datetimes).
> Containers were wrapped to expose a Python interface to an underlying C++
> container w/ no copying.
> 
> TLDR; Civilization IV shows an antipattern about how to embed Python in a
> native app and expecting anything resembling good performance. Civ IV used
> Boost Python, but they did it in an extremely poor, suboptimal fashion.
> Boost Python itself can be very performant, and I love the library. Boost
> Python, however does suffer from a learning curve that is akin to
> attempting to scale a 1000ft cliff with no rope or tools or even shoes. Due
> to the heavy use of template metaprogramming in C++, compiler errors if
> you're even just slightly off can be extremely hard to decipher. I've not
> used it lately or on modern compilers, so I don't know if it's gotten
> better, but tools like pretty_make.py (can find on github) can help filter
> out the compiler spewage from the real compiler error.
> 
> PS. I don't think that Python couldn't/shouldn't be used for scripting a
> game, I'm just offering up Civ IV as an example of how it can be done
> horribly wrong.
> 
> Regards,
> Nate
> 
> On Sun, Nov 13, 2016 at 7:03 PM, Chris Angelico <rosuav at gmail.com> wrote:
> 
> > On Mon, Nov 14, 2016 at 11:40 AM, Steve D'Aprano
> > <steve+python at pearwood.info> wrote:
> > > On Mon, 14 Nov 2016 12:23 am, Chris Angelico wrote:
> > >
> > >> Python without its stdlib is a disappointingly featureless
> > >> language :)
> > >
> > >
> > > I don't think that's true. You can do a lot in Python without any
> > imports in
> > > your code:
> > >
> > > - basic string processing
> > > - BigNum (int) and float arithmetic
> > > - lists and tuples
> > > - dicts
> > > - lazy processing of iterators
> > > - custom functions and classes
> > > - functional style zip, map and filter
> > >
> > > and more.
> > >
> > > Some parts of the standard library are required for built-in functions,
> > e.g.
> > > the io module is needed to open files. But I think you could probably
> > > reduce the standard library by, oh, 90% and still have a decent language
> > > for game scripting.
> >
> > Perhaps what would be more accurate is that Python without its stdlib
> > is as featureless as (say) Lua, but still without being safe to embed.
> > Yes, it's technically usable, but you really don't have a lot. And
> > remember, some of the stdlib is what we call builtins; if you embed
> > Python and engage "restricted mode", it's by emptying out the
> > builtins. You're pretty much down to three things:
> >
> > 1) Stuff that you could pass to ast.literal_eval
> > 2) Control flow statements
> > 3) Hacks that get you around the limitations, thus proving that
> > Python-without-stdlib is really really hard to actually test anything
> > with.
> >
> > ChrisA
> > --
> > https://mail.python.org/mailman/listinfo/python-list
> >


From steve+comp.lang.python at pearwood.info  Sun Nov 13 23:54:26 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Mon, 14 Nov 2016 15:54:26 +1100
Subject: if iter(iterator) is iterator
References: <f803560e-6af3-735e-735c-666437e1d5d0@rece.vub.ac.be>
 <mailman.129.1479067429.30068.python-list@python.org>
 <582907f5$0$1607$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmoOCnyRWg5FWUKrsWaqi5JFFXaA_Crb9NzxP7f7R=6WAg@mail.gmail.com>
 <mailman.134.1479085171.30068.python-list@python.org>
Message-ID: <58294384$0$1615$c3e8da3$5496439d@news.astraweb.com>

On Monday 14 November 2016 11:59, Chris Angelico wrote:

> On Mon, Nov 14, 2016 at 11:40 AM, Steve D'Aprano
> <steve+python at pearwood.info> wrote:
>> def isiterable(obj):
>>     """Return True if obj is an iterable, and False otherwise.
>>
>>     Iterable objects can be iterated over, and they provide either
>>     an __iter__ method or a __getitem__ method.
>>
>>     Iteration over an object tries calling the object's __iter__
>>     method (if any), then repeatedly calls __next__ on the result
>>     until a StopIteration exception is raised (the Iterator
>>     Protocol). Otherwise, it tries calling __getitem__ with
>>     arguments 0, 1, 2, 3 ... until an IndexError exception is
>>     raised (the Sequence Protocol).
>>
>>     """
>>     T = type(obj)
>>     return hasattr(T, '__iter__') or hasattr(T, '__getitem__')
> 
> Any particular reason to write it that way, rather than:
> 
> def isiterable(obj):
>     try:
>         iter(obj)
>         return True
>     except TypeError:
>         return False


class BadIterable:
    def __iter__(self):
        self.launch_missiles()
        return self


In general, merely testing whether something is a certain kind of thing 
shouldn't actually run that thing's code.

(That's not always possible. E.g. the thing's metaclass might launch missiles 
when you check for the existence of an attribute. But reducing the vulnerable 
surface as much as possible is a good thing.)



-- 
Steven
299792.458 km/s ? not just a good idea, it?s the law!



From greg.ewing at canterbury.ac.nz  Sun Nov 13 23:55:08 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Mon, 14 Nov 2016 17:55:08 +1300
Subject: Making stl files with python for 3d printing
In-Reply-To: <53f713f9-bf6e-4b58-869a-0b18f3c6bc2a@googlegroups.com>
References: <53f713f9-bf6e-4b58-869a-0b18f3c6bc2a@googlegroups.com>
Message-ID: <e8sudeF5munU1@mid.individual.net>

Poul Riis wrote:
> However, when sending the .stl file produced to a 3D-printer the vase comes
> out as a filled solid - no room for water and flowers!

My guess is that you have a problem with the orientation
of your faces. The inner surface needs to have its triangles
oriented so that their "outside" direction is towards the
centre of the vase.

If you generate the inner surface using the same code as
the outer surface and just change the radius, you will
end up with both surfaces oriented outwards, which would
produce the effect you describe.

-- 
Greg


From rosuav at gmail.com  Mon Nov 14 00:04:08 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 14 Nov 2016 16:04:08 +1100
Subject: if iter(iterator) is iterator
In-Reply-To: <58294384$0$1615$c3e8da3$5496439d@news.astraweb.com>
References: <f803560e-6af3-735e-735c-666437e1d5d0@rece.vub.ac.be>
 <mailman.129.1479067429.30068.python-list@python.org>
 <582907f5$0$1607$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmoOCnyRWg5FWUKrsWaqi5JFFXaA_Crb9NzxP7f7R=6WAg@mail.gmail.com>
 <mailman.134.1479085171.30068.python-list@python.org>
 <58294384$0$1615$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPTjJmpmhZeknasODYPx0kRTxMJKrfRzXywP4+S03oSxOExu0A@mail.gmail.com>

On Mon, Nov 14, 2016 at 3:54 PM, Steven D'Aprano
<steve+comp.lang.python at pearwood.info> wrote:
>> Any particular reason to write it that way, rather than:
>>
>> def isiterable(obj):
>>     try:
>>         iter(obj)
>>         return True
>>     except TypeError:
>>         return False
>
>
> class BadIterable:
>     def __iter__(self):
>         self.launch_missiles()
>         return self
>
>
> In general, merely testing whether something is a certain kind of thing
> shouldn't actually run that thing's code.

True, but ISTR there being a possibility that __iter__ could raise to
indicate that this isn't actually iterable. Maybe I'm recalling wrong.

ChrisA


From greg.ewing at canterbury.ac.nz  Mon Nov 14 00:18:12 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Mon, 14 Nov 2016 18:18:12 +1300
Subject: Best way to go about embedding python
In-Reply-To: <mailman.136.1479086155.30068.python-list@python.org>
References: <16cbd5d3-e2df-4635-bd4e-5772cb079c6f@googlegroups.com>
 <713d8eaf-030c-3874-19ee-7771545b1dbe@gmail.com>
 <mailman.136.1479086155.30068.python-list@python.org>
Message-ID: <e8svomF5vhiU1@mid.individual.net>

Michael Torrie wrote:
> And such bindings may need to be two-way.
> For example in Python code you may want to instantiate some new instance
> of game object, and you'll need to make sure that the wrapper code will
> create the appropriate in-game object.

Another thing that can be tricky about this is object
lifetimes. Python uses reference counting and garbage
collection to manage the lifetime of its data. If
your game engine has its own ideas on how to manage
lifetimes of its data structures, you may have
difficulties.

Typically you end up with a situation where each
game object has an associated "shadow" Python object,
with the two joined together like siamese twins.
You need to worry about keeping the Python object
alive as long as the game object is alive and vice
versa -- while making sure that they don't keep
each other alive forever.

If you have enough freedom, you may find it easier to
turn the problem around -- instead of embedding Python
in the game, embed the game in Python. Implement the
game engine as a Python extension module, where the
game objects are all inherently Python objects, and
Python is responsible for all of the lifetime management.
All the above headaches then go away, and a bonus you
get to write all the high-level logic of the game in Python.

-- 
Greg


From steve+comp.lang.python at pearwood.info  Mon Nov 14 00:20:49 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Mon, 14 Nov 2016 16:20:49 +1100
Subject: Access to the caller's globals, not your own
Message-ID: <582949b2$0$11109$c3e8da3@news.astraweb.com>

Suppose I have a library function that reads a global configuration setting:

# Toy example
SPAMIFY = True
def make_spam(n):
    if SPAMIFY:
        return "spam"*n
    else:
        return "ham"*n


Don't tell me to make SPAMIFY a parameter of the function. I know that. That's 
what I would normally do, but *occasionally* it is still useful to have a 
global configuration setting, and those are the cases I'm talking about.


Now the caller can say:

import library
library.SPAMIFY = False
result = library.make_spam(99)


but that would be Wrong and Bad and Evil, because you're affecting *other* 
code, not your code, which relies on SPAMIFY being True. Doing this is (maybe) 
okay when testing the library, but not for production use.

What I really want is for make_spam() to be able to look at the caller's 
globals, so that each caller can create their own per-module configuration 
setting:

import library
SPAMIFY = False  # only affects this module, no other modules
result = library.make_spam(99)


But... how can make_spam() look for the caller's globals rather than its own?

I'd rather not do this:


def make_spam(n, namespace):
    SPAMIFY = namespace['SPAMIFY']
    ...

which forces the user to do this:

result = library.make_spam(99, globals())


which defeats the purpose of making it a global setting. (If I wanted to 
*require* the caller to pass a parameter, I would just force them to pass in 
the flag itself. The whole point is to not require that.)

I'd be okay with making the namespace optional:


def make_spam(n, namespace=None):
    if namespace is None:
        namespace = ... # magic to get the caller's globals
    SPAMIFY = namespace['SPAMIFY']
    ...


but what magic do I need? globals() is no good, because it returns the 
library's global namespace, not the caller's.


Any solution ought to work for CPython, IronPython and Jython, at a minimum.




-- 
Steven
299792.458 km/s ? not just a good idea, it?s the law!



From eryksun at gmail.com  Mon Nov 14 00:55:49 2016
From: eryksun at gmail.com (eryk sun)
Date: Mon, 14 Nov 2016 05:55:49 +0000
Subject: Access to the caller's globals, not your own
In-Reply-To: <582949b2$0$11109$c3e8da3@news.astraweb.com>
References: <582949b2$0$11109$c3e8da3@news.astraweb.com>
Message-ID: <CACL+1asOTxe8=-GSB=j7kx9ksvMqD34Dwy9C-8nuAuuMwrZUZQ@mail.gmail.com>

On Mon, Nov 14, 2016 at 5:20 AM, Steven D'Aprano
<steve+comp.lang.python at pearwood.info> wrote:
> but what magic do I need? globals() is no good, because it returns the
> library's global namespace, not the caller's.
>
> Any solution ought to work for CPython, IronPython and Jython, at a minimum.

You can access the globals of the caller's frame, but you'll have to
research to what extent IronPython and Jython support CPython frame
objects.

FWIW:

    import inspect

    SPAMIFY = True

    def make_spam(n):
        caller_globals = inspect.currentframe().f_back.f_globals
        if caller_globals.get('SPAMIFY', SPAMIFY):
            return "spam" * n
        else:
            return "ham" * n

For example:

    >>> import library
    >>> library.make_spam(5)
    'spamspamspamspamspam'
    >>> SPAMIFY = False
    >>> library.make_spam(5)
    'hamhamhamhamham'


From torriem at gmail.com  Mon Nov 14 01:14:21 2016
From: torriem at gmail.com (Michael Torrie)
Date: Sun, 13 Nov 2016 23:14:21 -0700
Subject: Best way to go about embedding python
In-Reply-To: <c8ac8ef8-a323-4a26-8ed4-ae3458faa669@googlegroups.com>
References: <16cbd5d3-e2df-4635-bd4e-5772cb079c6f@googlegroups.com>
 <CAPTjJmpsYiBWYjMi2QKQmkTYG4Ag--m_f9n2BmWFFVWSyiQa+Q@mail.gmail.com>
 <mailman.127.1479043421.30068.python-list@python.org>
 <5829080f$0$1607$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmoFQoV+zsL5C8DzvXS98A89XLtbWhu44Prgt1ro+jAvnQ@mail.gmail.com>
 <CAPjBxg8GNqkLPqUSjomTEofk=RAff_kMKxYi3-vA+q2b0Xu9Vw@mail.gmail.com>
 <mailman.137.1479087490.30068.python-list@python.org>
 <c8ac8ef8-a323-4a26-8ed4-ae3458faa669@googlegroups.com>
Message-ID: <f9831de0-a659-6d22-ec94-97d17f962530@gmail.com>

On 11/13/2016 08:40 PM, Kaylen Wheeler wrote:
> I wonder if Lua would be a better option.  Does it suffer from the
> same sandboxing issues that python does?

Lua was designed for this sort of thing and it can be secured and sandboxed.


From steve+comp.lang.python at pearwood.info  Mon Nov 14 01:21:18 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Mon, 14 Nov 2016 17:21:18 +1100
Subject: if iter(iterator) is iterator
References: <f803560e-6af3-735e-735c-666437e1d5d0@rece.vub.ac.be>
 <mailman.129.1479067429.30068.python-list@python.org>
 <582907f5$0$1607$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmoOCnyRWg5FWUKrsWaqi5JFFXaA_Crb9NzxP7f7R=6WAg@mail.gmail.com>
 <mailman.134.1479085171.30068.python-list@python.org>
 <58294384$0$1615$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmpmhZeknasODYPx0kRTxMJKrfRzXywP4+S03oSxOExu0A@mail.gmail.com>
 <mailman.139.1479099852.30068.python-list@python.org>
Message-ID: <582957e0$0$1515$c3e8da3$5496439d@news.astraweb.com>

On Monday 14 November 2016 16:04, Chris Angelico wrote:

> True, but ISTR there being a possibility that __iter__ could raise to
> indicate that this isn't actually iterable. Maybe I'm recalling wrong.

If __iter__ raises, that's a bug :-)

Raising an exception from __iter__ appears to just fall through. There doesn't 
seem to be any special handling:



py> class X:
...     def __iter__(self):
...             raise TypeError("foo")
... 
py> iter(X())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in __iter__
TypeError: foo


Contrast that to actual non-iterables:

py> iter(23)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'int' object is not iterable



The way to indicate a class is not iterable is to not give it an __iter__ 
method in the first place.

If the class inherits __iter__, and you want to over-ride it, you might try 
setting the child class __iter__ to None. That trick works for __hash__. It 
doesn't work for __iter__ in Python 3.3, but it might work in more recent 
versions.



-- 
Steven
299792.458 km/s ? not just a good idea, it?s the law!



From storchaka at gmail.com  Mon Nov 14 01:25:07 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 14 Nov 2016 08:25:07 +0200
Subject: if iter(iterator) is iterator
In-Reply-To: <582907f5$0$1607$c3e8da3$5496439d@news.astraweb.com>
References: <f803560e-6af3-735e-735c-666437e1d5d0@rece.vub.ac.be>
 <mailman.129.1479067429.30068.python-list@python.org>
 <582907f5$0$1607$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o0blbv$vvr$1@blaine.gmane.org>

On 14.11.16 02:40, Steve D'Aprano wrote:
> I'm surprised that the inspect module doesn't appear to have isiterable and
> isiterator functions. Here's my first attempt at both:

Just use isinstance() with collections ABC classes.




From eryksun at gmail.com  Mon Nov 14 01:43:34 2016
From: eryksun at gmail.com (eryk sun)
Date: Mon, 14 Nov 2016 06:43:34 +0000
Subject: if iter(iterator) is iterator
In-Reply-To: <o0blbv$vvr$1@blaine.gmane.org>
References: <f803560e-6af3-735e-735c-666437e1d5d0@rece.vub.ac.be>
 <mailman.129.1479067429.30068.python-list@python.org>
 <582907f5$0$1607$c3e8da3$5496439d@news.astraweb.com>
 <o0blbv$vvr$1@blaine.gmane.org>
Message-ID: <CACL+1avKji4E=jDpMX42Ly7-i_556Q5QCW11+sOTF0KsW=YESQ@mail.gmail.com>

On Mon, Nov 14, 2016 at 6:25 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> On 14.11.16 02:40, Steve D'Aprano wrote:
>>
>> I'm surprised that the inspect module doesn't appear to have isiterable
>> and isiterator functions. Here's my first attempt at both:
>
> Just use isinstance() with collections ABC classes.

Except objects that use the legacy __getitem__ iterator aren't
instances of collections.Iterable:

    class C:
        def __getitem__(self, index):
            return index

    o = C()

    >>> isinstance(o, collections.Iterable)
    False
    >>> it = iter(o)
    >>> type(it)
    <class 'iterator'>
    >>> next(it), next(it), next(it)
    (0, 1, 2)


From al.johri at gmail.com  Mon Nov 14 03:51:49 2016
From: al.johri at gmail.com (Atul Johri)
Date: Mon, 14 Nov 2016 03:51:49 -0500
Subject: exit ThreadPoolExecutor immediately
Message-ID: <CAOFpNCc2A0bdC+JRyjGx_VVRgp4Vf+eoPHoEcSBryRy81s4s4Q@mail.gmail.com>

I am looking for a way to stop a ThreadPoolExecutor immediately under the
assumption that I don't care about what's currently running or pending.

```
limit = 2
executor = ThreadPoolExecutor(10)
posts = itertools.islice(mygen(executor=executor, **kwargs), 0, limit)
for post in posts:
  print(post)
executor.shutdown(wait=False)
```

Basically I have a generator, mygen, which is using the executor to submit
many tasks in parallel and yield the result one at a time. I would like to
be able to limit the generator and have the executor stop processing
immediately.

I was considering clearing the _work_queue or iterating over it and running
future.cancel() on each future but both seem to be quite hacky and didn't
work in my initial try.

https://github.com/python/cpython/blob/master/Lib/concurrent/futures/thread.py#L83

Any ideas?

- Al


From rgaddi at highlandtechnology.invalid  Mon Nov 14 12:57:27 2016
From: rgaddi at highlandtechnology.invalid (Rob Gaddi)
Date: Mon, 14 Nov 2016 17:57:27 -0000 (UTC)
Subject: Access to the caller's globals, not your own
References: <582949b2$0$11109$c3e8da3@news.astraweb.com>
Message-ID: <o0ctu7$ceg$1@dont-email.me>

Steven D'Aprano wrote:

> Suppose I have a library function that reads a global configuration setting:
>
> # Toy example
> SPAMIFY = True
> def make_spam(n):
>     if SPAMIFY:
>         return "spam"*n
>     else:
>         return "ham"*n
>
>
> Don't tell me to make SPAMIFY a parameter of the function. I know that. That's 
> what I would normally do, but *occasionally* it is still useful to have a 
> global configuration setting, and those are the cases I'm talking about.
>
>
> Now the caller can say:
>
> import library
> library.SPAMIFY = False
> result = library.make_spam(99)
>
>
> but that would be Wrong and Bad and Evil, because you're affecting *other* 
> code, not your code, which relies on SPAMIFY being True. Doing this is (maybe) 
> okay when testing the library, but not for production use.
>
> What I really want is for make_spam() to be able to look at the caller's 
> globals, so that each caller can create their own per-module configuration 
> setting:
>
> import library
> SPAMIFY = False  # only affects this module, no other modules
> result = library.make_spam(99)
>
>
> But... how can make_spam() look for the caller's globals rather than its own?
>
> I'd rather not do this:
>
>
> def make_spam(n, namespace):
>     SPAMIFY = namespace['SPAMIFY']
>     ...
>
> which forces the user to do this:
>
> result = library.make_spam(99, globals())
>
>
> which defeats the purpose of making it a global setting. (If I wanted to 
> *require* the caller to pass a parameter, I would just force them to pass in 
> the flag itself. The whole point is to not require that.)
>
> I'd be okay with making the namespace optional:
>
>
> def make_spam(n, namespace=None):
>     if namespace is None:
>         namespace = ... # magic to get the caller's globals
>     SPAMIFY = namespace['SPAMIFY']
>     ...
>
>
> but what magic do I need? globals() is no good, because it returns the 
> library's global namespace, not the caller's.
>
>
> Any solution ought to work for CPython, IronPython and Jython, at a minimum.
>

class Library:
  SPAMIFY = False
  def make_spam(m):
    ...

import library
Library = library.Library()

result = Library.make_spam(99)

-- 
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order.  See above to fix.


From 380162267qq at gmail.com  Mon Nov 14 13:44:27 2016
From: 380162267qq at gmail.com (380162267qq at gmail.com)
Date: Mon, 14 Nov 2016 10:44:27 -0800 (PST)
Subject: Why does this list swap fail?
Message-ID: <0a9a171f-9344-440f-9271-59372643468e@googlegroups.com>

L=[2,1]
L[0],L[L[0]-1]=L[L[0]-1],L[0]

The L doesn't change. Can someone provide me the detail procedure of this expression?


From jussi.piitulainen at helsinki.fi  Mon Nov 14 14:02:33 2016
From: jussi.piitulainen at helsinki.fi (Jussi Piitulainen)
Date: Mon, 14 Nov 2016 21:02:33 +0200
Subject: Why does this list swap fail?
References: <0a9a171f-9344-440f-9271-59372643468e@googlegroups.com>
Message-ID: <lf54m39lwae.fsf@ling.helsinki.fi>

380162267qq at gmail.com writes:

> L=[2,1]
> L[0],L[L[0]-1]=L[L[0]-1],L[0]
>
> The L doesn't change. Can someone provide me the detail procedure of
> this expression?

The right-hand side evaluates to (1,2), but then the assignments to the
targets on the left-hand side are processed in order from left to right.
This happens:

L[0] = 1
L[L0] - 1] = 2

https://docs.python.org/3/reference/simple_stmts.html#assignment-statements


From rgaddi at highlandtechnology.invalid  Mon Nov 14 14:07:36 2016
From: rgaddi at highlandtechnology.invalid (Rob Gaddi)
Date: Mon, 14 Nov 2016 19:07:36 -0000 (UTC)
Subject: Why does this list swap fail?
References: <0a9a171f-9344-440f-9271-59372643468e@googlegroups.com>
Message-ID: <o0d21o$t0q$1@dont-email.me>

380162267qq at gmail.com wrote:

> L=[2,1]
> L[0],L[L[0]-1]=L[L[0]-1],L[0]
>
> The L doesn't change. Can someone provide me the detail procedure of this expression?

Better question: Why does that list swap exist?  If you're just trying
to put the language through its paces that's one thing, but I certainly
hope that's not actually going to be production code.  Even if it were
right it's unreadable.

-- 
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order.  See above to fix.


From alain at universite-de-strasbourg.fr.invalid  Mon Nov 14 14:07:53 2016
From: alain at universite-de-strasbourg.fr.invalid (Alain Ketterlin)
Date: Mon, 14 Nov 2016 20:07:53 +0100
Subject: Why does this list swap fail?
References: <0a9a171f-9344-440f-9271-59372643468e@googlegroups.com>
Message-ID: <87d1hx6fsm.fsf@universite-de-strasbourg.fr.invalid>

380162267qq at gmail.com writes:

> L=[2,1]
> L[0],L[L[0]-1]=L[L[0]-1],L[0]
>
> The L doesn't change. Can someone provide me the detail procedure of
> this expression?

From:

https://docs.python.org/3/reference/simple_stmts.html#grammar-token-assignment_stmt

| Although the definition of assignment implies that overlaps between the
| left-hand side and the right-hand side are ?simultaneous? (for example
| a, b = b, a swaps two variables), overlaps within the collection of
| assigned-to variables occur left-to-right, sometimes resulting in
| confusion. For instance, the following program prints [0, 2]:
|
| x = [0, 1]
| i = 0
| i, x[i] = 1, 2         # i is updated, then x[i] is updated
| print(x)

In your case:

- L[0] get 1 (from L[L[0]-1])
- L[L[0]-1] (from lhs) is now L[0], which gets 2 (from L[0] in rhs)

Since both operations happen in sequence (not simultaneously), you just
overwrite the first element twice.

Congratulations, a very good corner case (you can't call it a bug, since
it conforms to the definition of the language).

-- Alain.


From ojacobson at heroku.com  Mon Nov 14 15:04:35 2016
From: ojacobson at heroku.com (ojacobson at heroku.com)
Date: Mon, 14 Nov 2016 12:04:35 -0800 (PST)
Subject: Access to the caller's globals, not your own
In-Reply-To: <582949b2$0$11109$c3e8da3@news.astraweb.com>
References: <582949b2$0$11109$c3e8da3@news.astraweb.com>
Message-ID: <a5e43dc5-0012-4c04-b0a3-0aba45c587cf@googlegroups.com>

On Monday, November 14, 2016 at 12:21:00 AM UTC-5, Steven D'Aprano wrote:

> Don't tell me to make SPAMIFY a parameter of the function. I know that. That's 
> what I would normally do, but *occasionally* it is still useful to have a 
> global configuration setting, and those are the cases I'm talking about.

This is the motivation behind Racket's parameters system <http://docs.racket-lang.org/reference/parameters.html>. A _parameter_ has the following properties:

* It can be read or written whenever it is in scope. By convention, parameters are global variables, but they can also be created in more limited lexical scopes.

* A parameter can be set. Once set, all subsequent reads in any scope will produce the last set value.

So far, so good: these are exactly like module-scoped globals in Python. Where it gets interesting is the parameterize function <http://docs.racket-lang.org/reference/parameters.html#%28form._%28%28lib._racket%2Fprivate%2Fmore-scheme..rkt%29._parameterize%29%29>: it introduces a dynamically-scoped set of new bindings for all included parameters, then evaluates a body (a procedure) with those bindings, then removes the bindings and restores the parameters to their values prior to parameterize. Any changes to those parameters within the body is also reverted: it's applied to the new binding introduced by parameterize, rather than to the "parent" binding.

Parameterizing a call means that changes to the "global variables" implemented as parameters have predictable scope and can be reliably restored to their prior values, meaning they're a fairly safe way to implement "config" globals.

Parameters are implemented using thread-local variables that point to stacks of bindings, where parameterize introduces a new stack frame for each listed parameter.

It's a neat system.

-o


From danpineau5 at gmail.com  Mon Nov 14 15:19:52 2016
From: danpineau5 at gmail.com (Dan Pineau)
Date: Mon, 14 Nov 2016 21:19:52 +0100
Subject: Fwd: Problems with Anaconda and Pyzo
In-Reply-To: <CAJSuWxwDdEj0_+BStmCgS-FAzZxmcCvwEs7Xfs0dWVe2qWpD6A@mail.gmail.com>
References: <CAJSuWxy9h98uYDZSKuz7Xvts==jDobV0sK5YjpbJqDmnpax-BA@mail.gmail.com>
 <CAJSuWxwDdEj0_+BStmCgS-FAzZxmcCvwEs7Xfs0dWVe2qWpD6A@mail.gmail.com>
Message-ID: <CAJSuWxxby8PNYfoAOW5PdctXEhSd5rtiAzp_U_3fLYPZAoD0mA@mail.gmail.com>

Hello,
I'm a french student and I'm running Python 3.5.2 on windows.

When I installed Anaconda3 4.2.0 version, Pyzo couldn't find a module named
conda.
Then I registered it as my default Python 3.5.2 version, but I couldn't use
Pyzo anymore (I sent you a screenshot). So I tried to fix this problem by
using the option "Repair" in the Python 3.5.2 setup, but this prevent me
from using Anaconda.

The problem might be simple, but I don't know what to do.
Could I have some help please ?

Have a good day.


From dfnsonfsduifb at gmx.de  Mon Nov 14 16:13:41 2016
From: dfnsonfsduifb at gmx.de (Johannes Bauer)
Date: Mon, 14 Nov 2016 22:13:41 +0100
Subject: Lexer/Parser question: TPG
Message-ID: <o0d9e5$6od$1@news.albasani.net>

Hi group,

this is not really a Python question, but I use Python to lex/parse some
input. In particular, I use the amazing TPG (http://cdsoft.fr/tpg/).
However, I'm now stuck at a point and am sure I'm not doing something
correctly -- since there's a bunch of really smart people here, I hope
to get some insights. Here we go:

I've created a minimal example in which I'm trying to parse some tokens
(strings and ints in the minimal example). Strings are delimited by
braces (). Therefore

(Foo) -> "Foo"

Braces inside braces are taken literally when balanced. If not balanced,
it's a parsing error.

(Foo (Bar)) -> "Foo (Bar)"

Braces may be escaped:

(Foo \)Bar) -> "Foo )Bar"

In my first (naive) attempt, I ignored the escaping and went with lexing
and then these rules:

token string_token '[^()]*';

[...]

String/s -> start_string                   $ s = ""
       (
           string_token/e                  $ s += e
           | String/e                      $ s += "(" + e + ")"
       )*
       end_string
       ;

While this worked a little bit (with some erroneous parsing,
admittedly), at least it *somewhat* worked. In my second attempt, I
tried to do it properly. I omitted the tokenization and instead used
inline terminals (which have precendence in TPG):

String/s -> start_string               $ s = ""
       (
           '\\.'/e                     $ s += "ESCAPED[" + e + "]"
           | '[^\\()]+'/e              $ s += e
           | String/e                  $ s += "(" + e + ")"
       )*
       end_string
       ;

(the "ESCAPED" part is just for demonstration to get the idea).

While the latter parser parses all strings perfectly, it now isn't able
to parse anything else anymore (including integer values!). Instead, it
appears to match the inline terminal '[^\\()]+' to my integer and then
dies (when trying, for example, to parse "12345"):

[  1][ 3]START.Expression.Value: (1,1) _tok_2 12345 != integer
[  2][ 3]START.Expression.String: (1,1) _tok_2 12345 != start_string
Traceback (most recent call last):
  File "example.py", line 56, in <module>
    print(Parser()(example))
  File "example/tpg.py", line 942, in __call__
    return self.parse('START', input, *args, **kws)
  File "example/tpg.py", line 1125, in parse
    return Parser.parse(self, axiom, input, *args, **kws)
  File "example/tpg.py", line 959, in parse
    value = getattr(self, axiom)(*args, **kws)
  File "<string>", line 3, in START
  File "<string>", line 14, in Expression
UnboundLocalError: local variable 'e' referenced before assignment

"_tok_2" seems to correspond to one of the inline terminal symbols, the
only one that fits would be '[^\\()]+'. But why would that *ever* match?
I thought it'd only match once a "start_string" was encountered (which
it isn't).

Since I'm the parsing noob, I don't think TPG (which is FREAKING
AMAZING, seriously!) is at fault but rather my understanding of TPG. Can
someone help me with this?

I've uploaded a complete working example to play around with here:

http://wikisend.com/download/642120/example.tar.gz
(if it's not working, please tell me and I'll look for some place else).

Thank you so much for your help,
Best regards,
Johannes

-- 
>> Wo hattest Du das Beben nochmal GENAU vorhergesagt?
> Zumindest nicht ?ffentlich!
Ah, der neueste und bis heute genialste Streich unsere gro?en
Kosmologen: Die Geheim-Vorhersage.
 - Karl Kaos ?ber R?diger Thomas in dsa <hidbv3$om2$1 at speranza.aioe.org>


From ned at nedbatchelder.com  Mon Nov 14 17:33:26 2016
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Mon, 14 Nov 2016 14:33:26 -0800 (PST)
Subject: help on "from deen import *" vs. "import deen"
In-Reply-To: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>
References: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>
Message-ID: <ca14fafb-2226-493e-ada5-e850a3c74f48@googlegroups.com>

On Sunday, November 13, 2016 at 9:39:12 PM UTC-5, jf... at ms4.hinet.net wrote:
> Running the following codes (deen.py) under Win32 python 3.4.4 terminal:
> 
> tbli = [0x66, 0x27, 0xD0]
> tblm = [0 for x in range(3)]
> def gpa(data):
>     td = data ^ tblm[2]
>     return td

The function gpa references the global tblm. Globals in Python are
global to the module (.py file) they appear in, and each module has
its own set of globals.  So the when the gpa function accesses tblm,
it's always going to be accessing the tblm in the deen module,
because gpa is defined in the deen module.


> 
> I can get a correct answer this way:
> >>> import deen
> >>> deen.tblm = deen.tbli
> >>> deen.gpa(0x7d)
> 173  # 0xad (= 0x7d ^ 0xd0) is decimal 173

Here you are updating deen.tblm, so when you call gpa, it sees the
new value you assigned to deen.tblm.

> 
> But I get a wrong answer this way:
> >>> from deen import *
> >>> tblm = tbli
> >>> gpa(0x7d)
> 125  # it's 0x7d, the tblm[2] is 0
> 
> Why? why! why:-(

Here you are assigning a value to your own tblm, not deen.tblm,
so gpa does not see the new value.

HTH,

--Ned.


From jfong at ms4.hinet.net  Mon Nov 14 20:36:47 2016
From: jfong at ms4.hinet.net (jfong at ms4.hinet.net)
Date: Mon, 14 Nov 2016 17:36:47 -0800 (PST)
Subject: help on "from deen import *" vs. "import deen"
In-Reply-To: <ca14fafb-2226-493e-ada5-e850a3c74f48@googlegroups.com>
References: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>
 <ca14fafb-2226-493e-ada5-e850a3c74f48@googlegroups.com>
Message-ID: <e0478664-9610-4d83-8f58-583682d11fcd@googlegroups.com>

Ned Batchelder at 2016/11/15 6:33:54AM wrote:
> > But I get a wrong answer this way:
> > >>> from deen import *
> > >>> tblm = tbli
> > >>> gpa(0x7d)
> > 125  # it's 0x7d, the tblm[2] is 0
> > 
> > Why? why! why:-(
> 
> Here you are assigning a value to your own tblm, not deen.tblm,
> so gpa does not see the new value.
> 
Hi Ned, below is the capture of my terminal screen:

D:\Temp>python
Python 3.4.4 (v3.4.4:737efcadf5a6, Dec 20 2015, 19:28:18) [MSC v.1600 32 bit (In
tel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> dir()
['__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__']

>>> from deen import *
>>> dir()
['__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__',
 'gpa', 'tbli', 'tblm']
>>> tblm
[0, 0, 0]
>>> tblm=tbli
>>> tblm
[102, 39, 208]
>>> dir()
['__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__',
 'gpa', 'tbli', 'tblm']
>>>

Do you mean the last 'tblm' is my own? How it can be? The 'tblm' already exist before the assignment and this assignment shouldn't create a new one, right? and how do I assign the 'deen.tblm' under this circumstance?

--Jach



From cfkaran2 at gmail.com  Mon Nov 14 22:15:19 2016
From: cfkaran2 at gmail.com (Cem Karan)
Date: Mon, 14 Nov 2016 22:15:19 -0500
Subject: Who owns the memory in ctypes?
Message-ID: <FBEF58C1-44A7-40E8-AA91-C7320570F5F9@gmail.com>

Hi all, I'm hoping that this will be an easy question.

I have a pile of C code that I wrote that I want to interface to via the ctypes module (https://docs.python.org/3/library/ctypes.html).  The C code uses the Boehm-Demers-Weiser garbage collector (http://www.hboehm.info/gc/) for all of its memory management.  What I want to know is, who owns allocated memory?  That is, if my C code allocates memory via GC_MALLOC() (the standard call for allocating memory in the garbage collector), and I access some object via ctypes in python, will the python garbage collector assume that it owns it and attempt to dispose of it when it goes out of scope?

Ideally, the memory is owned by the side that created it, with the other side simply referencing it, but I want to be sure before I invest a lot of time interfacing the two sides together.

Thanks,
Cem Karan

From python at mrabarnett.plus.com  Mon Nov 14 22:31:21 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 15 Nov 2016 03:31:21 +0000
Subject: help on "from deen import *" vs. "import deen"
In-Reply-To: <e0478664-9610-4d83-8f58-583682d11fcd@googlegroups.com>
References: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>
 <ca14fafb-2226-493e-ada5-e850a3c74f48@googlegroups.com>
 <e0478664-9610-4d83-8f58-583682d11fcd@googlegroups.com>
Message-ID: <bb816fbf-1df7-522d-bab7-9f7a65deaef3@mrabarnett.plus.com>

On 2016-11-15 01:36, jfong at ms4.hinet.net wrote:
> Ned Batchelder at 2016/11/15 6:33:54AM wrote:
>> > But I get a wrong answer this way:
>> > >>> from deen import *
>> > >>> tblm = tbli
>> > >>> gpa(0x7d)
>> > 125  # it's 0x7d, the tblm[2] is 0
>> >
>> > Why? why! why:-(
>>
>> Here you are assigning a value to your own tblm, not deen.tblm,
>> so gpa does not see the new value.
>>
> Hi Ned, below is the capture of my terminal screen:
>
> D:\Temp>python
> Python 3.4.4 (v3.4.4:737efcadf5a6, Dec 20 2015, 19:28:18) [MSC v.1600 32 bit (In
> tel)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
>>>> dir()
> ['__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__']
>
>>>> from deen import *
>>>> dir()
> ['__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__',
>  'gpa', 'tbli', 'tblm']
>>>> tblm
> [0, 0, 0]
>>>> tblm=tbli
>>>> tblm
> [102, 39, 208]
>>>> dir()
> ['__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__',
>  'gpa', 'tbli', 'tblm']
>>>>
>
> Do you mean the last 'tblm' is my own? How it can be? The 'tblm' already exist before the assignment and this assignment shouldn't create a new one, right? and how do I assign the 'deen.tblm' under this circumstance?
>
> --Jach
>
In the module 'deen' there's the name 'tblm' and the name 'tbli', among 
others:

     [module deen]

     tblm -----------> [0, 0, 0]

     tbli -----------> [102, 39, 208]

When you say "from deen import *" you're copying names and their 
references from the module's namespace to your local namespace:

     [module deen]                                   [locally]

     tblm ------------> [0, 0, 0] <----------------- tblm

     tbli ------------> [102, 39, 208] <------------ tbli

When you say "tblm = ..." you're binding the local name to a different 
object, so when you say "tblm = tbli" you get:

     [module deen]                                   [locally]

     tblm ------------> [0, 0, 0]    :-------------- tblm
                                     V
     tbli ------------> [102, 39, 208] <------------ tbli



From dan at tombstonezero.net  Mon Nov 14 23:55:22 2016
From: dan at tombstonezero.net (Dan Sommers)
Date: Tue, 15 Nov 2016 04:55:22 -0000 (UTC)
Subject: Access to the caller's globals, not your own
References: <582949b2$0$11109$c3e8da3@news.astraweb.com>
Message-ID: <o0e4fp$tjb$1@dont-email.me>

On Mon, 14 Nov 2016 16:20:49 +1100, Steven D'Aprano wrote:

> import library
> SPAMIFY = False  # only affects this module, no other modules
> result = library.make_spam(99)

I must be missing something, because it seems too obvious:

    import library
    # only affects this module, no other modules
    library = library.make_library(spamify=False)
    # ...
    result = library.make_spam(99)

And then in library.py:

    class Library:
        def__init__(self, spamify=False):
            self.spamify = spamify
        def make_spam(self, p):
            return ((p, "spammified")
                    if self.spamify else
                    (p, "hammified"))

    def make_library(spamify=False):
        return Library(spamify)

How do you want the following code to work:

    import library
    SPAMIFY=False

    def make_false_spam():
        return library.make_spam(99)

    def make_true_spam():
        global SPAMIFY
        SPAMIFY=True
        return library.make_spam(99)

I don't have to tell you how many things can go wrong with code like
that.  ;-)

Yes, it's a straw man.  No, I don't think we have all the details of
your use case.  Yes, I'm willing to have missed something subtle (or not
so subtle).

Dan


From eryksun at gmail.com  Tue Nov 15 00:19:46 2016
From: eryksun at gmail.com (eryk sun)
Date: Tue, 15 Nov 2016 05:19:46 +0000
Subject: Who owns the memory in ctypes?
In-Reply-To: <FBEF58C1-44A7-40E8-AA91-C7320570F5F9@gmail.com>
References: <FBEF58C1-44A7-40E8-AA91-C7320570F5F9@gmail.com>
Message-ID: <CACL+1atj9EzW9fQOWKnrZxFFGKdrCMwGeTcDH6TAEMYhriY+ew@mail.gmail.com>

On Tue, Nov 15, 2016 at 3:15 AM, Cem Karan <cfkaran2 at gmail.com> wrote:
> if my C code allocates memory via GC_MALLOC() (the standard call for allocating memory
> in the garbage collector), and I access some object via ctypes in python, will the python
> garbage collector assume that it owns it and attempt to dispose of it when it goes out of
> scope?

ctypes objects own only the memory that they allocate. Inspect the
_b_needsfree_ attribute to determine whether a ctypes object owns the
referenced memory.

For example:

This array object owns a 12-byte buffer for the array elements:

    >>> arr = (ctypes.c_uint * 3)(0, 1, 2)
    >>> arr._b_needsfree_
    1

This pointer object owns an 8-byte buffer for the 64-bit target address:

    >>> p = ctypes.POINTER(ctypes.c_uint * 3)(arr)
    >>> p._b_needsfree_
    1

The following new array object created by dereferencing the pointer
does not own the 12-byte buffer:

    >>> ref = p[0]
    >>> ref[:]
    [0, 1, 2]
    >>> ref._b_needsfree_
    0

However, it does have a reference chain back to the original array:

    >>> ref._b_base_ is p
    True
    >>> p._objects['1'] is arr
    True

On the other hand, if you use the from_address() class method, the
resulting object is a dangling reference to memory that it doesn't own
and for which there's no supporting reference chain to the owning
object.

For example:

    >>> arr = (ctypes.c_uint * 2**24)()
    >>> arr[-1] = 42
    >>> ref = type(arr).from_address(ctypes.addressof(arr))
    >>> ref[-1]
    42

    >>> ref._b_base_ is None
    True
    >>> ref._objects is None
    True

2**24 bytes is a big allocation that uses mmap (Unix) instead of the
heap. Thus accessing ref[-1] causes a segfault after arr is
deallocated:

    >>> del arr
    >>> ref[-1]
    Segmentation fault (core dumped)


From jfong at ms4.hinet.net  Tue Nov 15 01:32:54 2016
From: jfong at ms4.hinet.net (jfong at ms4.hinet.net)
Date: Mon, 14 Nov 2016 22:32:54 -0800 (PST)
Subject: help on "from deen import *" vs. "import deen"
In-Reply-To: <mailman.147.1479180690.30068.python-list@python.org>
References: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>
 <ca14fafb-2226-493e-ada5-e850a3c74f48@googlegroups.com>
 <e0478664-9610-4d83-8f58-583682d11fcd@googlegroups.com>
 <bb816fbf-1df7-522d-bab7-9f7a65deaef3@mrabarnett.plus.com>
 <mailman.147.1479180690.30068.python-list@python.org>
Message-ID: <1a6f991d-53c1-4c79-b81b-e804b01be653@googlegroups.com>

MRAB at 2016/11/15 11:31:41AM wrote:
> When you say "from deen import *" you're copying names and their 
> references from the module's namespace to your local namespace:
> 
>      [module deen]                                   [locally]
> 
>      tblm ------------> [0, 0, 0] <----------------- tblm
> 
>      tbli ------------> [102, 39, 208] <------------ tbli

hmm...I always thought that the picture of "from deen import *" is

                                [locally]

     [0, 0, 0] <----------------- tblm

     [102, 39, 208] <------------ tbli

But obviously it's not. The compiled code of function gpa is still reference to a hidden deen.tblm 'global' object which I can't access anymore. A bad news:-(

--Jach



From dieter at handshake.de  Tue Nov 15 03:15:35 2016
From: dieter at handshake.de (dieter)
Date: Tue, 15 Nov 2016 09:15:35 +0100
Subject: exit ThreadPoolExecutor immediately
References: <CAOFpNCc2A0bdC+JRyjGx_VVRgp4Vf+eoPHoEcSBryRy81s4s4Q@mail.gmail.com>
Message-ID: <87zil1gnvc.fsf@handshake.de>

Atul Johri <al.johri at gmail.com> writes:
> I am looking for a way to stop a ThreadPoolExecutor immediately under the
> assumption that I don't care about what's currently running or pending.
>
> ```
> limit = 2
> executor = ThreadPoolExecutor(10)
> posts = itertools.islice(mygen(executor=executor, **kwargs), 0, limit)
> for post in posts:
>   print(post)
> executor.shutdown(wait=False)
> ```
>
> Basically I have a generator, mygen, which is using the executor to submit
> many tasks in parallel and yield the result one at a time. I would like to
> be able to limit the generator and have the executor stop processing
> immediately.
>
> I was considering clearing the _work_queue or iterating over it and running
> future.cancel() on each future but both seem to be quite hacky and didn't
> work in my initial try.
>
> https://github.com/python/cpython/blob/master/Lib/concurrent/futures/thread.py#L83
>
> Any ideas?

I would implement my own class (inheriting from "ThreadPoolExecutor")
and give it appropriate API to realise what I need (using whatever
internal implementation details are necessary).

I should not be that difficult for tasks not yet started (i.e.
not yet run be a thread).
In earlier Python versions, it was not
possible from Python level to abort a thread; there was a C level
Python API function to abort a thread once it started to execute Python
code again. Thus, there was not a complete solution for your problem.
This might have changed with modern Python version, but I doubt it
(the main reason for the restriction has been the difficulty
to cleanly abort a thread in a platform independent way -- this
difficulty should remain, whatever the Python version).



From torriem at gmail.com  Tue Nov 15 09:43:32 2016
From: torriem at gmail.com (Michael Torrie)
Date: Tue, 15 Nov 2016 07:43:32 -0700
Subject: help on "from deen import *" vs. "import deen"
In-Reply-To: <1a6f991d-53c1-4c79-b81b-e804b01be653@googlegroups.com>
References: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>
 <ca14fafb-2226-493e-ada5-e850a3c74f48@googlegroups.com>
 <e0478664-9610-4d83-8f58-583682d11fcd@googlegroups.com>
 <bb816fbf-1df7-522d-bab7-9f7a65deaef3@mrabarnett.plus.com>
 <mailman.147.1479180690.30068.python-list@python.org>
 <1a6f991d-53c1-4c79-b81b-e804b01be653@googlegroups.com>
Message-ID: <144b0566-6738-bd37-cd1c-1585e70761d2@gmail.com>

On 11/14/2016 11:32 PM, jfong at ms4.hinet.net wrote:
> But obviously it's not. The compiled code of function gpa is still
> reference to a hidden deen.tblm 'global' object which I can't access
> anymore. A bad news:-(

Seems like you're still not understanding Python variables.

After importing all the names from "deen" into your current module's
namespace, they do "point" or refer to the actual objects from the deen
module. So at that moment, they are the same objects as if you were to
import deen and refer to them with the deen.foo notation. However once
you assign to these names in your current module you are breaking this
link to the deen module and assigning a new object to these names in
your current module's namespace.  You're getting hung up on the names,
when you should be concentrating on what the names refer to.

Variables in python aren't implement the same as C, or Java. They aren't
aliases for a box in memory that you can alter using assignment.
Instead assignment points the name to the object in the RHS, in place of
the old object it used to refer to (if the references to this old object
reach zero, it is deleted).

As you've been told several times, if you "import deen" then you can
place a new object into the deen namespace using something like:

deen.foo=bar

Importing everything from an imported module into the current module's
namespace is not the best idea.


From woooee at gmail.com  Tue Nov 15 11:03:53 2016
From: woooee at gmail.com (woooee at gmail.com)
Date: Tue, 15 Nov 2016 08:03:53 -0800 (PST)
Subject: exit ThreadPoolExecutor immediately
In-Reply-To: <mailman.144.1479145031.30068.python-list@python.org>
References: <CAOFpNCc2A0bdC+JRyjGx_VVRgp4Vf+eoPHoEcSBryRy81s4s4Q@mail.gmail.com>
 <mailman.144.1479145031.30068.python-list@python.org>
Message-ID: <1cb6a88a-083d-4d5b-8ce4-25b9dc460a4b@googlegroups.com>

Doug Hellmann has what looks like a similar example using a poison pill (2nd example at) https://pymotw.com/2/multiprocessing/communication.html#multiprocessing-queues  Note that join() allows the processes to finish so it you don't care then don't "join()" them.  You can also terminate a multiprocessing thread https://pymotw.com/2/multiprocessing/basics.html#terminating-processes


From ygutfreund at gmail.com  Tue Nov 15 11:36:26 2016
From: ygutfreund at gmail.com (ygutfreund at gmail.com)
Date: Tue, 15 Nov 2016 08:36:26 -0800 (PST)
Subject: Python Guarded Pattern matching (when x -> do action)
Message-ID: <81a4fea5-a66d-4a11-807b-cee969a0b23a@googlegroups.com>

I am looking to see if there is prior work, or design ideas for implementing pattern-matched guard statements in a very natural format for python programmers.

For those not familiar with pattern matching in [SCALA][1], [Erlang][2], or [F#][3]

They all have constructs similiar to:

When (pattern == condition) then invoke( behavior)

This is also very similiar to trigger-stored-procedure techniques in Databases.

What we want to do is create an agent-based system, where there is a large shared memory, with lots of binary structs (python has good modules for this). Which has many attached guard-statement rules, that will fire when the values match the pattern (we will have monitors that watch when structs are modified).

Has anyone seen prior work of this nature? The goal is to get something that is quite natural to a python programmer, not force SCALA, GO, Erlang, F#, etc on python folks.

We have our own ideas, but I would love to see what the community might have already done along these lines.


  [1]: http://docs.scala-lang.org/tutorials/tour/pattern-matching.html
  [2]: http://stackoverflow.com/questions/21083874/erlang-case-statement
  [3]: https://docs.microsoft.com/en-us/dotnet/articles/fsharp/language-reference/match-expressions


From lucaberto at libero.it  Tue Nov 15 12:01:07 2016
From: lucaberto at libero.it (luca72)
Date: Tue, 15 Nov 2016 09:01:07 -0800 (PST)
Subject: PyQt pass data from class
Message-ID: <997899a9-fdfa-47e9-9203-a4e4b2acd4a4@googlegroups.com>

Hello i need to this

class Form(QWidget, Ui_Form):
    """
    Class documentation goes here.
    """
    def __init__(self, parent=None):
        """
        Constructor
        
        @param parent reference to the parent widget
        @type QWidget
        """
        super(Form, self).__init__(parent)
        self.setupUi(self)
        self.tabWidget.setCurrentIndex(0)
        combo = QComboBox()
        self.disegno = Cornice()

class Cornice(QPainter):
    
    def __init__(self, parent=None):
        
        lista_dati = []

in the class Form i have a lineEdit

in the class Cornice i need to write something link

self.lineEdit.setText('blabla') that is in the class Form

in wich way i can have access to the lineedit of class Form without event from class Cornice

Many Thanks


From rgaddi at highlandtechnology.invalid  Tue Nov 15 12:40:40 2016
From: rgaddi at highlandtechnology.invalid (Rob Gaddi)
Date: Tue, 15 Nov 2016 17:40:40 -0000 (UTC)
Subject: PyQt pass data from class
References: <997899a9-fdfa-47e9-9203-a4e4b2acd4a4@googlegroups.com>
Message-ID: <o0fhao$hho$1@dont-email.me>

luca72 wrote:

> Hello i need to this
>
> class Form(QWidget, Ui_Form):
>     """
>     Class documentation goes here.
>     """
>     def __init__(self, parent=None):
>         """
>         Constructor
>         
>         @param parent reference to the parent widget
>         @type QWidget
>         """
>         super(Form, self).__init__(parent)
>         self.setupUi(self)
>         self.tabWidget.setCurrentIndex(0)
>         combo = QComboBox()
>         self.disegno = Cornice()
>
> class Cornice(QPainter):
>     
>     def __init__(self, parent=None):
>         
>         lista_dati = []
>
> in the class Form i have a lineEdit
>
> in the class Cornice i need to write something link
>
> self.lineEdit.setText('blabla') that is in the class Form
>
> in wich way i can have access to the lineedit of class Form without event from class Cornice
>
> Many Thanks

Traditionally you'd have Cornice emit a Signal, which when you
__init__the Form you'd connect to the appropriate slot.  I don't recall
PyQt syntax, but in PySide it would look like

	self.disengo.theSignal.connect(self.lineEdit.setText)


-- 
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order.  See above to fix.


From torriem at gmail.com  Tue Nov 15 13:41:53 2016
From: torriem at gmail.com (Michael Torrie)
Date: Tue, 15 Nov 2016 11:41:53 -0700
Subject: PyQt pass data from class
In-Reply-To: <997899a9-fdfa-47e9-9203-a4e4b2acd4a4@googlegroups.com>
References: <997899a9-fdfa-47e9-9203-a4e4b2acd4a4@googlegroups.com>
Message-ID: <8e2c50ec-03e5-7171-eb24-06da664fc637@gmail.com>

On 11/15/2016 10:01 AM, luca72 via Python-list wrote:
> in wich way i can have access to the lineedit of class Form without event from class Cornice

If I understand you, you are asking how to set the text without having
it emit a signal. Is that correct? Or are you asking how to access the
member of one class from a method in another class?

If it's the former, you can ask PyQt to suppress a callback for a while.
 Look up the blockSignals() method of QWidget.

If it's the latter, your best bet is to pass the object you want to
access into the other class somehow, perhaps as an argument to __init__().



From priisdk at gmail.com  Tue Nov 15 13:44:52 2016
From: priisdk at gmail.com (Poul Riis)
Date: Tue, 15 Nov 2016 10:44:52 -0800 (PST)
Subject: Making stl files with python for 3d printing
In-Reply-To: <e8sudeF5munU1@mid.individual.net>
References: <53f713f9-bf6e-4b58-869a-0b18f3c6bc2a@googlegroups.com>
 <e8sudeF5munU1@mid.individual.net>
Message-ID: <8a7cf49f-73dc-4b9f-b6cd-9729f8c924ab@googlegroups.com>

Den mandag den 14. november 2016 kl. 05.55.20 UTC+1 skrev Gregory Ewing:
> Poul Riis wrote:
> > However, when sending the .stl file produced to a 3D-printer the vase comes
> > out as a filled solid - no room for water and flowers!
> 
> My guess is that you have a problem with the orientation
> of your faces. The inner surface needs to have its triangles
> oriented so that their "outside" direction is towards the
> centre of the vase.
> 
> If you generate the inner surface using the same code as
> the outer surface and just change the radius, you will
> end up with both surfaces oriented outwards, which would
> produce the effect you describe.
> 
> -- 
> Greg

You are right - thank you so much!

Poul Riis


From lake.honrich at gmail.com  Tue Nov 15 16:57:14 2016
From: lake.honrich at gmail.com (shadecelebi)
Date: Tue, 15 Nov 2016 13:57:14 -0800 (PST)
Subject: A question about sprite rendering in game development
Message-ID: <746749d6-342d-43f2-ae48-8ea2a5118133@googlegroups.com>

I'm still quite new to python, but I have a concern. I want to learn python for game development, but I need to know if python is capable of rendering, let's say 50 animated character sprites, at once without lagging. 

if anyone's ever played the epic war game series then you should have an idea of what I'm trying to create. in short, I need several different types of characters to be under the players command while there are several of each character and each of them also need to be able to react to their envoirment (being able to notice when an opponent is in front of them) 

is all of this possible in python without massive lagg??


From marko at pacujo.net  Tue Nov 15 17:13:56 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Wed, 16 Nov 2016 00:13:56 +0200
Subject: A question about sprite rendering in game development
References: <746749d6-342d-43f2-ae48-8ea2a5118133@googlegroups.com>
Message-ID: <877f84gzmj.fsf@elektro.pacujo.net>

shadecelebi <lake.honrich at gmail.com>:

> I'm still quite new to python, but I have a concern. I want to learn
> python for game development, but I need to know if python is capable
> of rendering, let's say 50 animated character sprites, at once without
> lagging.
>
> if anyone's ever played the epic war game series then you should have
> an idea of what I'm trying to create. in short, I need several
> different types of characters to be under the players command while
> there are several of each character and each of them also need to be
> able to react to their envoirment (being able to notice when an
> opponent is in front of them)
>
> is all of this possible in python without massive lagg??

Your question suggests Python is the least of your problems. Do you
already have the artwork for even one of your game characters?

Anyway, do create your game in Python. You should at least be able to
test your game ideas with it. If it is too slow, simply reimplement your
prototype in C. Compared with everything else, learning Python and
reimplementing the game in C will be trivial.


Marko


From python at lucidity.plus.com  Tue Nov 15 17:16:07 2016
From: python at lucidity.plus.com (Erik)
Date: Tue, 15 Nov 2016 22:16:07 +0000
Subject: help on "from deen import *" vs. "import deen"
In-Reply-To: <144b0566-6738-bd37-cd1c-1585e70761d2@gmail.com>
References: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>
 <ca14fafb-2226-493e-ada5-e850a3c74f48@googlegroups.com>
 <e0478664-9610-4d83-8f58-583682d11fcd@googlegroups.com>
 <bb816fbf-1df7-522d-bab7-9f7a65deaef3@mrabarnett.plus.com>
 <mailman.147.1479180690.30068.python-list@python.org>
 <1a6f991d-53c1-4c79-b81b-e804b01be653@googlegroups.com>
 <144b0566-6738-bd37-cd1c-1585e70761d2@gmail.com>
Message-ID: <9b6ad48d-ee65-bca6-6fa7-4176ddaae83e@lucidity.plus.com>

On 15/11/16 14:43, Michael Torrie wrote:
> As you've been told several times, if you "import deen" then you can
> place a new object into the deen namespace using something like:
>
> deen.foo=bar
>
> Importing everything from an imported module into the current module's
> namespace is not the best idea

But "from foo import *" is not importing "everything". It's the opposite 
- it's importing everything _that the module wants to explicitly expose_ 
(i.e., a subset of everything ;)).

It *used* to be everything in the module's namespace, but then the 
possibly misnamed "__all__" magic variable made it a subset of whatever 
the module's author wanted to expose.

However, "import foo" _does_ import "everything", and also gives the 
importer the power to re-bind names within that module's namespace too 
(which is the crux of the original question). This is not usually a good 
thing unless the importing module is incestuous with the imported module.

So this brings up another point - not sure if it has been made before: 
should a module have a way of *not* allowing an importer to re-bind its 
local bindings?

For example, something like a "__bindable__" variable such that all 
names not included may not be bound by any other module? And/or 
something like an "__export__" variable that says what gets exposed to a 
simple "import foo" (sometimes one might want to "import foo" and 
sometimes "from foo import *" for namespace reasons, but why should 
those two statements import different things)?

E.


From steve+python at pearwood.info  Tue Nov 15 17:37:53 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Wed, 16 Nov 2016 09:37:53 +1100
Subject: A question about sprite rendering in game development
References: <746749d6-342d-43f2-ae48-8ea2a5118133@googlegroups.com>
Message-ID: <582b8e43$0$1588$c3e8da3$5496439d@news.astraweb.com>

On Wed, 16 Nov 2016 08:57 am, shadecelebi wrote:

> I'm still quite new to python, but I have a concern. I want to learn
> python for game development, but I need to know if python is capable of
> rendering, let's say 50 animated character sprites, at once without
> lagging.

Absolutely. For something like that, I suggest you look at PyGame. You do
all the game programming and logic in Python, and the underlying game
engine (including drawing the sprites) is based on fast C and assembly
code. It supports multiple backends, including opengl, directx, windib,
X11, linux frame buffer, and even ASCII art.

http://www.pygame.org/
http://www.pygame.org/wiki/about



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Tue Nov 15 19:33:10 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Wed, 16 Nov 2016 11:33:10 +1100
Subject: help on "from deen import *" vs. "import deen"
References: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>
 <ca14fafb-2226-493e-ada5-e850a3c74f48@googlegroups.com>
 <e0478664-9610-4d83-8f58-583682d11fcd@googlegroups.com>
 <bb816fbf-1df7-522d-bab7-9f7a65deaef3@mrabarnett.plus.com>
 <mailman.147.1479180690.30068.python-list@python.org>
 <1a6f991d-53c1-4c79-b81b-e804b01be653@googlegroups.com>
 <144b0566-6738-bd37-cd1c-1585e70761d2@gmail.com>
 <9b6ad48d-ee65-bca6-6fa7-4176ddaae83e@lucidity.plus.com>
 <mailman.153.1479248175.30068.python-list@python.org>
Message-ID: <582ba948$0$1589$c3e8da3$5496439d@news.astraweb.com>

On Wed, 16 Nov 2016 09:16 am, Erik wrote:

> On 15/11/16 14:43, Michael Torrie wrote:
>> As you've been told several times, if you "import deen" then you can
>> place a new object into the deen namespace using something like:
>>
>> deen.foo=bar
>>
>> Importing everything from an imported module into the current module's
>> namespace is not the best idea
> 
> But "from foo import *" is not importing "everything". It's the opposite
> - it's importing everything _that the module wants to explicitly expose_
> (i.e., a subset of everything ;)).

I think you're skipping Michael's point and making a pedantic comment that
isn't really relevant and, to be even more pedantic, is wrong.

The gory details of what precisely gets imported by an import star is
probably not relevant to a beginner still struggling with understanding
fundamental scoping issues.

I'm also sure that Michael didn't *literally* mean "everything", since he's
been around a while and is probably aware that private names with a single
leading underscore won't be imported.

And your comment is (pedantically) wrong because it isn't always the case
that "import *" only imports things that the module explicitly exposes for
importing. By default, it imports everything which isn't implicitly
excluded. Modules don't pre-define an __all__ variable, hence modules have
to opt out of allowing imports of all non-private names rather than opt in.

I'm sure this is all important for intermediate-level Python programmers,
but I just hope we're not confusing the OP.


> It *used* to be everything in the module's namespace, but then the
> possibly misnamed "__all__" magic variable made it a subset of whatever
> the module's author wanted to expose.

It isn't misnamed: it's intended as a shorter version of "all public names".
The intent being, module authors list ALL their module's public names in
__all__ in order to control what import star does, without relying purely
on the implicit _private name convention.


> However, "import foo" _does_ import "everything", 

No, that's a misuse of terminology. It only imports a single thing: `foo`.


You can test this yourself by comparing the local namespace before and after
a single import: only one more name is added to your namespace.

py> before = after = None
py> before = set(locals().keys())
py> assert 'cmath' not in before
py> import cmath
py> after = set(locals().keys())
py> after - before
{'cmath'}


The conclusion is obvious: importing a single name imports only a single
object into the current namespace. (Well, what else did you expect? *wink*)

What you are talking about is something different: importing a module gives
you access to all the module's attributes. Well of course it does. If you
import a single class from a module, having access to the class gives you
access to all the class attributes. Importing a single function gives you
access to the function's attributes. Importing any object at all gives you
access to that object's attributes. Modules are no different.


> and also gives the 
> importer the power to re-bind names within that module's namespace too
> (which is the crux of the original question). This is not usually a good
> thing unless the importing module is incestuous with the imported module.

With great power comes great responsibility... 

Monkey-patching modules is a powerful and risky technique. No *library*
should monkey-patch another library (unless they're designed to work
together), since you cannot tell if you're messing with something a third
library relies on. But its acceptable for an application to take over
control of a library and monkey-patch it, since there should only ever be a
single application running at once.

Perhaps only *borderline* acceptable, and maybe not something many people
are willing to do in production systems, but its certainly something we do
in (for example) testing. What is a test suite but a stand-alone
application? And test suites will often monkey-patch libraries, insert
mocks and stubs and otherwise manipulate the library.

At least we're not Ruby :-)

http://www.virtuouscode.com/2008/02/23/why-monkeypatching-is-destroying-ruby/


> So this brings up another point - not sure if it has been made before:
> should a module have a way of *not* allowing an importer to re-bind its
> local bindings?

Certainly not! Well, maybe. Yes.

Never mind this "the importer" bit, that's just a special case of a more
general problem: Python has no way of enforcing constants.

This has been argued about many times. Some people want a way of getting
real constants which cannot be modified, or at least cannot be re-bound to
a new value. Others think that the status quo ("if you don't want to
re-bind a constant, then don't re-bind it") is satisfactory.

I'd prefer to have a way to protect against *accidental* re-bindings:

const x = 1.2345
# later:
x += 1  # raise an exception

while still allowing some mechanism for *deliberate* monkey-patching, say:

vars()['x'] = 2.2345  # bypass const mechanism


Possibly something along the lines of property for modules might work. There
are tricks and hacks to get something along these lines already, usually
involving a module replacing itself with the instance of some other class,
but none of them are exactly easy, straightforward or obvious.



> For example, something like a "__bindable__" variable such that all
> names not included may not be bound by any other module? 

You're basically asking for a special case of "protected" and "private" for
classes, except you only want it to apply to modules. Seems weird.

In any case, the usual argument against these things (including the "const"
idea) is that We're All Adults Here. Developers of languages with strong
protections of this sort invariably end up spending large amounts of time
trying to bypass those features, which suggests that they're often
anti-features, or at least too easy to abuse.

Python takes a simpler approach: simple naming conventions (ALLCAPS for
constants, leading _underscore for private names) and trust the caller to
follow the convention.

> And/or 
> something like an "__export__" variable that says what gets exposed to a
> simple "import foo" (sometimes one might want to "import foo" and
> sometimes "from foo import *" for namespace reasons, but why should
> those two statements import different things)?

Because that's what they are designed to do.

`import foo` imports the module foo, that is all. (To be pedantic: it is
*nominally* a module. By design, it could be any object at all.)

`from foo import *` imports all the visible public attributes of foo.

They do completely different things, equivalent to something similar to:

five = int('5')


versus:


_tmp = int('5')
for name in dir(_tmp):
    if not name.startswith('_'):
        locals()[name] = getattr(_tmp, name)
del _tmp



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From jfong at ms4.hinet.net  Tue Nov 15 21:39:25 2016
From: jfong at ms4.hinet.net (jfong at ms4.hinet.net)
Date: Tue, 15 Nov 2016 18:39:25 -0800 (PST)
Subject: help on "from deen import *" vs. "import deen"
In-Reply-To: <mailman.151.1479221023.30068.python-list@python.org>
References: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>
 <ca14fafb-2226-493e-ada5-e850a3c74f48@googlegroups.com>
 <e0478664-9610-4d83-8f58-583682d11fcd@googlegroups.com>
 <bb816fbf-1df7-522d-bab7-9f7a65deaef3@mrabarnett.plus.com>
 <mailman.147.1479180690.30068.python-list@python.org>
 <1a6f991d-53c1-4c79-b81b-e804b01be653@googlegroups.com>
 <144b0566-6738-bd37-cd1c-1585e70761d2@gmail.com>
 <mailman.151.1479221023.30068.python-list@python.org>
Message-ID: <80870bc5-5975-43e0-bf23-dccb1f628d6f@googlegroups.com>

Michael Torrie at 2016/11/15 10:43:58PM wrote:
> Seems like you're still not understanding Python variables.

I do, just not get used to it yet:-)

> .... However once
> you assign to these names in your current module you are breaking this
> link to the deen module and assigning a new object to these names in
> your current module's namespace.

Why? the name "tblm" is already exist and is a mutable one, should be changed intuitional by "tblm=tbli". If I really need a new list, I can create one by using a new name. Python shouldn't have its finger in my pie:-(

> You're getting hung up on the names,
> when you should be concentrating on what the names refer to.

That's one problem I was concerned. Human beings are very deeply binding on words (names?). We think in words, talk in words, communicate in words. Actually we are living in words. I really don't like that when I said "John is a boy" and was told "No, John is a dog now":-)

--Jach


From torriem at gmail.com  Tue Nov 15 22:14:47 2016
From: torriem at gmail.com (Michael Torrie)
Date: Tue, 15 Nov 2016 20:14:47 -0700
Subject: help on "from deen import *" vs. "import deen"
In-Reply-To: <80870bc5-5975-43e0-bf23-dccb1f628d6f@googlegroups.com>
References: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>
 <ca14fafb-2226-493e-ada5-e850a3c74f48@googlegroups.com>
 <e0478664-9610-4d83-8f58-583682d11fcd@googlegroups.com>
 <bb816fbf-1df7-522d-bab7-9f7a65deaef3@mrabarnett.plus.com>
 <mailman.147.1479180690.30068.python-list@python.org>
 <1a6f991d-53c1-4c79-b81b-e804b01be653@googlegroups.com>
 <144b0566-6738-bd37-cd1c-1585e70761d2@gmail.com>
 <mailman.151.1479221023.30068.python-list@python.org>
 <80870bc5-5975-43e0-bf23-dccb1f628d6f@googlegroups.com>
Message-ID: <5d48ae5f-17f1-fca7-1239-d22c2950363c@gmail.com>

On 11/15/2016 07:39 PM, jfong at ms4.hinet.net wrote:
> Michael Torrie at 2016/11/15 10:43:58PM wrote:
>> Seems like you're still not understanding Python variables.
> 
> I do, just not get used to it yet:-)

No you don't yet.  See below.

> Why? the name "tblm" is already exist and is a mutable one, should be
> changed intuitional by "tblm=tbli". If I really need a new list, I
> can create one by using a new name. Python shouldn't have its finger
> in my pie:-(

No, names are not mutable per se.  Objects can be.  If the object is
mutable you can mutate it:

someobject.some_mutate_method(foo)

or

some_list[1] = 5

The latter is actually syntactic sugar for something more akin to this:
some_list.__setattr__(1,5).  Note that this throws out the reference to
the object that used to be held by slot 1, and now refers to the new
object, 5.

Names are, well, just names.  They refer to objects, which may or may
not be mutable.  And multiple names can refer to the exact same object.
Internally, variables are stored in dictionaries:

>>> a = 5
>>> globals['a']
5

And in fact in Python many primitive objects like numbers and strings
are immutable.  They cannot ever change once they are brought into
existence.  Expressions such as mathematical equations result in new
objects being created (number objects can be reused and are often cached).

The assignment operator does not by default mutate an object.  Instead
it makes the variable (the name) refer to the new object, the result of
the RHS.  Now this may seem like I've contradicted myself, seeing as I
used the example above to show how = can mutate a list-like object.  But
if you understand the underlying mechanism for holding names, it's
actually consistent.  The globals object is a dictionary and is itself
mutable.  But when we assign a new object to a particular dictionary
key, it tosses out the old reference and makes the key now refer to the
new object.  It does not do anything to the old object itself.

Anyway, don't think we can make things much clearer than that.

>> You're getting hung up on the names, when you should be
>> concentrating on what the names refer to.
> 
> That's one problem I was concerned. Human beings are very deeply
> binding on words (names?). We think in words, talk in words,
> communicate in words. Actually we are living in words. I really don't
> like that when I said "John is a boy" and was told "No, John is a dog
> now":-)

Not quite sure where you're going with that. I would think the idea of
labels on objects would be fairly natural. I could stick a label that
says "chair" on a chair, and then later move that label to the couch.
Same label, different object.  Whether the label makes sense is up to you.



From vek.m1234 at gmail.com  Wed Nov 16 00:21:03 2016
From: vek.m1234 at gmail.com (Veek M)
Date: Wed, 16 Nov 2016 10:51:03 +0530
Subject: __debug__
 http://stackoverflow.com/questions/15305688/conditional-debug-statement-not-executed-though-debug-is-true
Message-ID: <o0gqb1$l93$1@dont-email.me>

Trying to make sense of that article. My understanding of debug was 
simple:
1. __debug__ is always True, unless -O or -OO
2. 'if' is optimized out when True and the expr is inlined.

So what does he mean by:

1. 'If you rebind __debug__, it can cause symptoms'
2. 'During module compilation, the same code that handles literals also 
handles the magic constants ..., None, True, False, and __debug__'
3. 'you'll see that if __debug__: statements are either removed 
entirely, or use LOAD_CONST to load the compile-time debug constant, 
while if bool(__debug__): statements use LOAD_GLOBAL to load the value 
of __debug__.'

4. 'Of course these are guaranteed to be the same? unless you rebind 
__debug__'

Basically every line in that answer is new to me..


From vek.m1234 at gmail.com  Wed Nov 16 00:23:22 2016
From: vek.m1234 at gmail.com (Veek M)
Date: Wed, 16 Nov 2016 10:53:22 +0530
Subject: __debug__
 http://stackoverflow.com/questions/15305688/conditional-debug-statement-not-executed-though-debug-is-true
References: <o0gqb1$l93$1@dont-email.me>
Message-ID: <o0gqfa$l93$2@dont-email.me>

Veek M wrote:

> Trying to make sense of that article. My understanding of debug was
> simple:
> 1. __debug__ is always True, unless -O or -OO
> 2. 'if' is optimized out when True and the expr is inlined.
> 
> So what does he mean by:
> 
> 1. 'If you rebind __debug__, it can cause symptoms'
> 2. 'During module compilation, the same code that handles literals
> also handles the magic constants ..., None, True, False, and
> __debug__' 3. 'you'll see that if __debug__: statements are either
> removed entirely, or use LOAD_CONST to load the compile-time debug
> constant, while if bool(__debug__): statements use LOAD_GLOBAL to load
> the value of __debug__.'
> 
> 4. 'Of course these are guaranteed to be the same? unless you rebind
> __debug__'
> 
> Basically every line in that answer is new to me..
Sorry for the awful title but I was not sure what to title it because I 
have no clue what they are talking about.. barring the fact that it's 
about __debug__ and assigning True/False to it..

Isn't that how you turn it off when you don't want -O ??


From lucaberto at libero.it  Wed Nov 16 02:47:26 2016
From: lucaberto at libero.it (luca72)
Date: Tue, 15 Nov 2016 23:47:26 -0800 (PST)
Subject: PyQt pass data from class
In-Reply-To: <mailman.152.1479235319.30068.python-list@python.org>
References: <997899a9-fdfa-47e9-9203-a4e4b2acd4a4@googlegroups.com>
 <8e2c50ec-03e5-7171-eb24-06da664fc637@gmail.com>
 <mailman.152.1479235319.30068.python-list@python.org>
Message-ID: <302abcc5-1566-4d95-8417-c698a394f2e7@googlegroups.com>

Thanks for your reply

Is the latter, can you explain how i can do it.

Thanks


From dieter at handshake.de  Wed Nov 16 03:01:33 2016
From: dieter at handshake.de (dieter)
Date: Wed, 16 Nov 2016 09:01:33 +0100
Subject: __debug__
 http://stackoverflow.com/questions/15305688/conditional-debug-statement-not-executed-though-debug-is-true
References: <o0gqb1$l93$1@dont-email.me>
Message-ID: <877f837t0i.fsf@handshake.de>

Veek M <vek.m1234 at gmail.com> writes:

> Trying to make sense of that article. My understanding of debug was 
> simple:
> 1. __debug__ is always True, unless -O or -OO
> 2. 'if' is optimized out when True and the expr is inlined.
>
> So what does he mean by:
>
> 1. 'If you rebind __debug__, it can cause symptoms'
> 2. 'During module compilation, the same code that handles literals also 
> handles the magic constants ..., None, True, False, and __debug__'
> 3. 'you'll see that if __debug__: statements are either removed 
> entirely, or use LOAD_CONST to load the compile-time debug constant, 
> while if bool(__debug__): statements use LOAD_GLOBAL to load the value 
> of __debug__.'
>
> 4. 'Of course these are guaranteed to be the same? unless you rebind 
> __debug__'
>
> Basically every line in that answer is new to me..

It essentially tells you:
 
  "__debug__" has an internal use; only read it; never write (rebind) it.

The rest are the details explaining what can go wrong when you
write (rebind) "__debug__" and why.

The "why" comes essentially from the fact that in some (but not all)
cases "__debug__" is handled at compile time, while a potential
writing it handled at runtime. Thus, after you have changed "__debug__",
things may not work as you expect.


If you keep in mind "do not change the value of __debug__ at runtime", you
can forget all the details.



From steve+comp.lang.python at pearwood.info  Wed Nov 16 03:09:59 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Wed, 16 Nov 2016 19:09:59 +1100
Subject: urllib.request giving unexpected results
Message-ID: <582c1459$0$1500$c3e8da3$5496439d@news.astraweb.com>

I'm trying to download a file using urllib.request and pipe it straight to an 
external process. On Linux systems, the following is a test file that 
demonstrates the problem:


--- cut ---

#!/usr/bin/python3.5

import urllib.request
import subprocess

TEST_URL = 'https://www.irs.gov/pub/irs-prior/f1040--1864.pdf'

with urllib.request.urlopen(TEST_URL) as f:
    data = subprocess.check_output(['file', '-'], stdin=f)
    print(data)

with urllib.request.urlopen(TEST_URL) as f:
    with open('/tmp/x.pdf', 'wb') as g:
        n = g.write(f.read())
    with open('/tmp/x.pdf') as g:
        data = subprocess.check_output(['file', '-'], stdin=g)
    print(data)

--- cut ---

Output is:

b'/dev/stdin: data\n'
b'/dev/stdin: PDF document, version 1.6\n'


Expected output is:

b'/dev/stdin: PDF document, version 1.6\n'
b'/dev/stdin: PDF document, version 1.6\n'


If I just read from urllib.request, I get what appears to the naked eye to be 
the expected data:


py> with urllib.request.urlopen(TEST_URL) as f:
...     file = f.read()
... 
py> print(file[:100])
b'%PDF-1.6\r%\xe2\xe3\xcf\xd3\r\n55 0 obj\r<</Linearized 1/L 66721/O 57/E 
28286/N 4/T 65574/H [ 856 317]>>\rendobj\r    '




Certainly looks like a PDF file. So what's going on?




-- 
Steven
299792.458 km/s ? not just a good idea, it?s the law!



From steve+comp.lang.python at pearwood.info  Wed Nov 16 03:13:19 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Wed, 16 Nov 2016 19:13:19 +1100
Subject: Access to the caller's globals, not your own
References: <582949b2$0$11109$c3e8da3@news.astraweb.com>
 <CACL+1asOTxe8=-GSB=j7kx9ksvMqD34Dwy9C-8nuAuuMwrZUZQ@mail.gmail.com>
 <mailman.140.1479102994.30068.python-list@python.org>
Message-ID: <582c1520$0$1500$c3e8da3$5496439d@news.astraweb.com>

On Monday 14 November 2016 16:55, eryk sun wrote:

> On Mon, Nov 14, 2016 at 5:20 AM, Steven D'Aprano
> <steve+comp.lang.python at pearwood.info> wrote:
>> but what magic do I need? globals() is no good, because it returns the
>> library's global namespace, not the caller's.
>>
>> Any solution ought to work for CPython, IronPython and Jython, at a minimum.
> 
> You can access the globals of the caller's frame, but you'll have to
> research to what extent IronPython and Jython support CPython frame
> objects.
> 
> FWIW:
> 
>     import inspect
> 
>     SPAMIFY = True
> 
>     def make_spam(n):
>         caller_globals = inspect.currentframe().f_back.f_globals
>         if caller_globals.get('SPAMIFY', SPAMIFY):
>             return "spam" * n
>         else:
>             return "ham" * n

Nice! That appears to work on Jython, but not IronPython.

Thanks for the suggestion.





-- 
Steven
299792.458 km/s ? not just a good idea, it?s the law!



From steve+comp.lang.python at pearwood.info  Wed Nov 16 03:15:17 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Wed, 16 Nov 2016 19:15:17 +1100
Subject: Access to the caller's globals, not your own
References: <582949b2$0$11109$c3e8da3@news.astraweb.com>
 <a5e43dc5-0012-4c04-b0a3-0aba45c587cf@googlegroups.com>
Message-ID: <582c1596$0$1500$c3e8da3$5496439d@news.astraweb.com>

On Tuesday 15 November 2016 07:04, ojacobson at heroku.com wrote:

> On Monday, November 14, 2016 at 12:21:00 AM UTC-5, Steven D'Aprano wrote:
> 
>> Don't tell me to make SPAMIFY a parameter of the function. I know that.
>> That's what I would normally do, but *occasionally* it is still useful to
>> have a global configuration setting, and those are the cases I'm talking
>> about.
> 
> This is the motivation behind Racket's parameters system
> <http://docs.racket-lang.org/reference/parameters.html>. A _parameter_ has
> the following properties:
[snip]
> Parameterizing a call means that changes to the "global variables"
> implemented as parameters have predictable scope and can be reliably restored
> to their prior values, meaning they're a fairly safe way to implement
> "config" globals.

Sounds more like Python's "with" syntax and context managers, but I'll have a 
read over it the docs and see.


-- 
Steven
299792.458 km/s ? not just a good idea, it?s the law!



From lake.honrich at gmail.com  Wed Nov 16 03:16:30 2016
From: lake.honrich at gmail.com (shadecelebi)
Date: Wed, 16 Nov 2016 00:16:30 -0800 (PST)
Subject: A question about sprite rendering in game development
In-Reply-To: <746749d6-342d-43f2-ae48-8ea2a5118133@googlegroups.com>
References: <746749d6-342d-43f2-ae48-8ea2a5118133@googlegroups.com>
Message-ID: <0c4cb2a3-8678-40ad-b966-ab7c985b8875@googlegroups.com>

thanx a lot you guys. I'm slightly familiar with pygame from before so I'll make sure to utilize it. and no I don't have any of the characters yet as I've yet to start. I just wanted to know if I should keep learning python or if it would be trivial to try making my game a reality with this language. I'll need to learn a couple more things before I try to make a prototype. I believe the easiest route will be to simply rectangels of different colors for the early gameplay and to try and perfect the game mechanics before I move over to adding sprites and such. thank you both. I'll post updates in a different thread when I'm starting to see some kind of results


From rosuav at gmail.com  Wed Nov 16 03:24:21 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 16 Nov 2016 19:24:21 +1100
Subject: urllib.request giving unexpected results
In-Reply-To: <582c1459$0$1500$c3e8da3$5496439d@news.astraweb.com>
References: <582c1459$0$1500$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPTjJmrqNAdW8aosr9YNdgdj8WWB=7jKMUz5_2kei3qrttJxdw@mail.gmail.com>

On Wed, Nov 16, 2016 at 7:09 PM, Steven D'Aprano
<steve+comp.lang.python at pearwood.info> wrote:
> I'm trying to download a file using urllib.request and pipe it straight to an
> external process. On Linux systems, the following is a test file that
> demonstrates the problem:
>
>
> --- cut ---
>
> #!/usr/bin/python3.5
>
> import urllib.request
> import subprocess
>
> TEST_URL = 'https://www.irs.gov/pub/irs-prior/f1040--1864.pdf'
>
> with urllib.request.urlopen(TEST_URL) as f:
>     data = subprocess.check_output(['file', '-'], stdin=f)
>     print(data)

Interesting.

rosuav at sikorsky:~$ python3
Python 3.7.0a0 (default:72e64fc8746b+, Oct 28 2016, 12:35:28)
[GCC 6.2.0 20161010] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib.request
>>> import subprocess
>>> TEST_URL = 'https://www.irs.gov/pub/irs-prior/f1040--1864.pdf'
>>> with urllib.request.urlopen(TEST_URL) as f:
...    data = subprocess.check_output(['tee', 'tmp/asdfasdf'], stdin=f)
...

rosuav at sikorsky:~/tmp$ hd asdfasdf |head
00000000  17 03 03 40 18 e9 b0 79  7c 03 c8 5d 21 40 2f 11  |... at ...y|..]!@/.|
00000010  4a a3 f1 4d e0 19 04 fc  42 84 d9 cf 59 0b f8 56  |J..M....B...Y..V|
00000020  7d 35 08 88 17 50 24 8c  26 fe d8 13 2b fd 14 55  |}5...P$.&...+..U|
00000030  16 81 c3 1e 13 ae 00 1d  d4 8e 9f 0f a4 19 bb 44  |...............D|
00000040  46 d5 bf 25 28 d0 b0 23  44 6f 1c ef 84 d9 82 9b  |F..%(..#Do......|
00000050  17 15 3a 11 e1 ec de 59  65 d7 ea 41 dc 53 07 70  |..:....Ye..A.S.p|
00000060  99 d5 11 75 b7 90 7e cd  46 b5 67 ee 9a 62 18 63  |...u..~.F.g..b.c|
00000070  36 7f 7b df a1 fb 6d b8  66 8b 2f 82 e6 05 7e aa  |6.{...m.f./...~.|
00000080  d7 9f 9e 05 cf 06 68 6b  c8 4c df 5e 24 9d 92 f6  |......hk.L.^$...|
00000090  3d 53 76 11 c1 70 05 14  94 e5 5b ec b0 cf 64 70  |=Sv..p....[...dp|


So that's what file(1) is seeing. My guess is that a urlopen object
isn't "file-like" enough for subprocess. Maybe it's showing a more
"raw" version?

ChrisA


From steve+comp.lang.python at pearwood.info  Wed Nov 16 03:36:02 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Wed, 16 Nov 2016 19:36:02 +1100
Subject: Access to the caller's globals, not your own
References: <582949b2$0$11109$c3e8da3@news.astraweb.com>
 <o0e4fp$tjb$1@dont-email.me>
Message-ID: <582c1a73$0$11119$c3e8da3@news.astraweb.com>

On Tuesday 15 November 2016 15:55, Dan Sommers wrote:

> On Mon, 14 Nov 2016 16:20:49 +1100, Steven D'Aprano wrote:
> 
>> import library
>> SPAMIFY = False  # only affects this module, no other modules
>> result = library.make_spam(99)
> 
> I must be missing something, because it seems too obvious:
[snip]

I wouldn't say "obvious" so much as "complex".

A factory function or a class that holds state for a local make_spam() callable 
would be a possible solution, but it significantly increases the complexity, 
especially for the simple case:

import library
result = library.make_spam(arg)


versus:

import library
make_spam = library.make_library()
result = make_spam(arg)

What a drag.



[...]
> How do you want the following code to work:
> 
>     import library
>     SPAMIFY=False
> 
>     def make_false_spam():
>         return library.make_spam(99)
> 
>     def make_true_spam():
>         global SPAMIFY
>         SPAMIFY=True
>         return library.make_spam(99)
> 
> I don't have to tell you how many things can go wrong with code like
> that.  ;-)

Indeed.

> Yes, it's a straw man.  No, I don't think we have all the details of
> your use case.  Yes, I'm willing to have missed something subtle (or not
> so subtle).

The use-case shouldn't matter, because I think I've been hanging around here 
for long enough that you should trust me that I'm not one to abuse global 
variables :-)

But, for what its worth... 

I have a function, let's call it quartile(data, n). Unfortunately there's about 
a dozen different ways to calculate quartiles, and they all give ever-so-
slightly different results. So I end up with:

def quartile(data, n, scheme="Method 7"):
    ...


which lets the caller choose the specific calculation method they want to use, 
or just use the default. *My* default, chosen by me, which may or may not be 
convenient.

And then there's a bunch of other functions which depend on quartile(). I don't 
have to list them, but there's at least two and may be more. They too will have 
an optional parameter to choose a calculation method. And *that* almost rules 
out your factory function idea: having to call a factory to create multiple 
functions is too confusing for a library API.

    quartile, iqr, blah blah blah = library.make_functions()


So while I'd consider that under some circumstances, I don't like it here.

I'd like the caller to be able to set their own default, if they don't like 
mine, in the easiest way possible. That means a global.

DEFAULT_SCHEME = "Method 7"

# Late binding of the default, instead of early binding.
def quartile(data, n, scheme=None):
    if scheme is None:
        scheme = DEFAULT_SCHEME
    ...


Say they want to match the same calculation method that Excel uses, they can 
just modify the library.DEFAULT_SCHEME and forget all about it.

But that's globally global, and I actually want it to only be global to their 
own module. Hence my question.

As an alternative, I'm thinking of allowing the caller to set an environment 
variable... only that's globally global too.


-- 
Steven
299792.458 km/s ? not just a good idea, it?s the law!



From steve+comp.lang.python at pearwood.info  Wed Nov 16 03:39:38 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Wed, 16 Nov 2016 19:39:38 +1100
Subject: __debug__
 http://stackoverflow.com/questions/15305688/conditional-debug-statement-not-executed-though-debug-is-true
References: <o0gqb1$l93$1@dont-email.me>
Message-ID: <582c1b4a$0$11119$c3e8da3@news.astraweb.com>

On Wednesday 16 November 2016 16:21, Veek M wrote:

> Trying to make sense of that article. My understanding of debug was
> simple:
> 1. __debug__ is always True, unless -O or -OO
> 2. 'if' is optimized out when True and the expr is inlined.
> 
> So what does he mean by:
> 
> 1. 'If you rebind __debug__, it can cause symptoms'

What he means is, "I didn't test this code before running it, and I am wrong."

You cannot rebind __debug__.

>>> __debug__ = False
  File "<stdin>", line 1
SyntaxError: can not assign to __debug__


(That's Python 2.5 or better, and maybe even older than that.)


-- 
Steven
299792.458 km/s ? not just a good idea, it?s the law!



From ethan at stoneleaf.us  Wed Nov 16 04:30:22 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 16 Nov 2016 01:30:22 -0800
Subject: Access to the caller's globals, not your own
In-Reply-To: <582c1a73$0$11119$c3e8da3@news.astraweb.com>
References: <582949b2$0$11109$c3e8da3@news.astraweb.com>
 <o0e4fp$tjb$1@dont-email.me> <582c1a73$0$11119$c3e8da3@news.astraweb.com>
Message-ID: <582C272E.6080308@stoneleaf.us>

On 11/16/2016 12:36 AM, Steven D'Aprano wrote:

> But, for what its worth...
>
> I have a function, let's call it quartile(data, n). Unfortunately there's about
> a dozen different ways to calculate quartiles, and they all give ever-so-
> slightly different results. So I end up with:
>
> def quartile(data, n, scheme="Method 7"):
>      ...
>
>
> which lets the caller choose the specific calculation method they want to use,
> or just use the default. *My* default, chosen by me, which may or may not be
> convenient.
>
> And then there's a bunch of other functions which depend on quartile(). I don't
> have to list them, but there's at least two and may be more. They too will have
> an optional parameter to choose a calculation method. And *that* almost rules
> out your factory function idea: having to call a factory to create multiple
> functions is too confusing for a library API.
>
>      quartile, iqr, blah blah blah = library.make_functions()
>
>
> So while I'd consider that under some circumstances, I don't like it here.
>
> I'd like the caller to be able to set their own default, if they don't like
> mine, in the easiest way possible. That means a global.
>
> DEFAULT_SCHEME = "Method 7"
>
> # Late binding of the default, instead of early binding.
> def quartile(data, n, scheme=None):
>      if scheme is None:
>          scheme = DEFAULT_SCHEME
>      ...
>
>
> Say they want to match the same calculation method that Excel uses, they can
> just modify the library.DEFAULT_SCHEME and forget all about it.
>
> But that's globally global, and I actually want it to only be global to their
> own module. Hence my question.

I think you stuck with the same solution Decimal uses: a context that you can either set globally global, or one that can be used as a context manager.

--
~Ethan~


From mal at europython.eu  Wed Nov 16 05:21:25 2016
From: mal at europython.eu (M.-A. Lemburg)
Date: Wed, 16 Nov 2016 11:21:25 +0100
Subject: Farewell to Rob Collins
Message-ID: <582C3325.2020107@europython.eu>

We would like to share with you the sad news, that Rob Collins has
passed away earlier this month, on November 2nd, after a short but
intense illness.

Many of you may know Rob from the sponsored massage sessions he
regularly ran at EuroPython in recent years and which he continued to
develop, taking them from a single man setup (single threaded process)
to a group of people setup by giving workshops (multiprocessing) and
later on by passing on his skills to more leaders (removing the GIL)
to spread wellness and kindness throughout our conference series:

https://ep2015.europython.eu/conference/talks/sponsored-massage-training-in-aid-of-the-python-software-foundation

His massages regularly raised more than a thousand dollars which were
donated to the Python Software Foundation (PSF) to do even more good.

Rob also gave a lot of thoughtful talks at the EuroPython conferences,
always very cheerful, full of humor and many good insights. Here?s a
selection:

 * EuroPython 2015: DumbDev ? eight rules for dumb development
   https://www.youtube.com/watch?v=HEOI8y0qHMk

 * EuroPython 2015: Lightning Talk announcing the Python Massage
   Foundation
   https://youtu.be/WmvTfUYJ2Bw?t=1539

 * EuroPython 2013: TDM Test Driven Madness
   https://www.youtube.com/watch?v=Nj4nwh_VrPM

 * EuroPython 2012: An introduction to his neck and and shoulder
   massages
   https://www.youtube.com/watch?v=itn8W9zI0Wk

 * EuroPython 2011: Pricing products using Python graphs and sets
   https://www.youtube.com/watch?v=TN9nIBxDXU8

Rob was a true Pythonista from the heart. He will always be remembered
for his humor, great spirit and kindness.

You were such a cheerful person. We will miss you, Rob.

Thank you for all the inspiration,
-?
Your friends from the EuroPython community
http://www.europython-society.org/

Full blog post:
http://www.europython-society.org/post/153253946400/farewell-to-rob-collins


From kliateni at gmail.com  Wed Nov 16 06:04:09 2016
From: kliateni at gmail.com (Karim)
Date: Wed, 16 Nov 2016 12:04:09 +0100
Subject: Farewell to Rob Collins
In-Reply-To: <582C3325.2020107@europython.eu>
References: <582C3325.2020107@europython.eu>
Message-ID: <dac4b600-8d33-c446-bb72-ef20646224d4@gmail.com>


On 16/11/2016 11:21, M.-A. Lemburg wrote:
> s a true Pythonista from the heart. He will always be remember

RIP Rob.

Karim



From torriem at gmail.com  Wed Nov 16 10:01:34 2016
From: torriem at gmail.com (Michael Torrie)
Date: Wed, 16 Nov 2016 08:01:34 -0700
Subject: PyQt pass data from class
In-Reply-To: <302abcc5-1566-4d95-8417-c698a394f2e7@googlegroups.com>
References: <997899a9-fdfa-47e9-9203-a4e4b2acd4a4@googlegroups.com>
 <8e2c50ec-03e5-7171-eb24-06da664fc637@gmail.com>
 <mailman.152.1479235319.30068.python-list@python.org>
 <302abcc5-1566-4d95-8417-c698a394f2e7@googlegroups.com>
Message-ID: <cadcc497-82f2-ede5-ebe5-c15a1929d972@gmail.com>

On 11/16/2016 12:47 AM, luca72 via Python-list wrote:
> Thanks for your reply
> 
> Is the latter, can you explain how i can do it.

Just add an argument to __init__() of the one class where you will pass
the instance of the other class to it. Then you can store it inside the
instance, and use it whenever needed.

class foo(object):
    def __init__(self):
	pass

    def some_method(self):
        print ("Hello from foo.some_method")

class bar(object):
    def __init__(self, foo_instance):
        self.foo_instance = foo_instance

    def some_method(self):
        foo.some_method()

a=foo()
b=bar(a)

Of course since you're dealing with PyQt your classes will involve some
other parameters to __init__ but you can still tack your parameter on there.

You may want to re-examine how you're building your classes, though.
There's likely a way you can structure things to eliminate this kind of
coupling, especially in the context of a GUI. Often times making your
own signal that can be externally connected to a method of the other
class is the way to go as this reduces the coupling between the classes.



From antoon.pardon at rece.vub.ac.be  Wed Nov 16 10:19:02 2016
From: antoon.pardon at rece.vub.ac.be (Antoon Pardon)
Date: Wed, 16 Nov 2016 16:19:02 +0100
Subject: Access to the caller's globals, not your own
In-Reply-To: <582c1a73$0$11119$c3e8da3@news.astraweb.com>
References: <582949b2$0$11109$c3e8da3@news.astraweb.com>
 <o0e4fp$tjb$1@dont-email.me> <582c1a73$0$11119$c3e8da3@news.astraweb.com>
Message-ID: <97e9d179-7f11-5070-ef0f-2e839e5f44c4@rece.vub.ac.be>

Op 16-11-16 om 09:36 schreef Steven D'Aprano:
> On Tuesday 15 November 2016 15:55, Dan Sommers wrote:
>
>> On Mon, 14 Nov 2016 16:20:49 +1100, Steven D'Aprano wrote:
>>
>>> import library
>>> SPAMIFY = False  # only affects this module, no other modules
>>> result = library.make_spam(99)
>> I must be missing something, because it seems too obvious:
> [snip]
>
> I wouldn't say "obvious" so much as "complex".
>
> A factory function or a class that holds state for a local make_spam() callable 
> would be a possible solution, but it significantly increases the complexity, 
> especially for the simple case:
>
> import library
> result = library.make_spam(arg)
>
>
> versus:
>
> import library
> make_spam = library.make_library()
> result = make_spam(arg)
>
> What a drag.

What about how the random module solves this?

The random module provides a factory class: Random.

It also provides functions which are methods of a
hidden instantiation.

-- 
Antoon Pardon.



From eryksun at gmail.com  Wed Nov 16 10:22:21 2016
From: eryksun at gmail.com (eryk sun)
Date: Wed, 16 Nov 2016 15:22:21 +0000
Subject: __debug__ http://stackoverflow.com/questions/15305688
Message-ID: <CACL+1asTO3yF-y7E5JLUFfMrJ-NKC=ZCWt15AM6y0EKo4Z3MyA@mail.gmail.com>

On Wed, Nov 16, 2016 at 8:39 AM, Steven D'Aprano
<steve+comp.lang.python at pearwood.info> wrote:
> On Wednesday 16 November 2016 16:21, Veek M wrote:
>
>> Trying to make sense of that article. My understanding of debug was
>> simple:
>> 1. __debug__ is always True, unless -O or -OO
>> 2. 'if' is optimized out when True and the expr is inlined.
>>
>> So what does he mean by:
>>
>> 1. 'If you rebind __debug__, it can cause symptoms'
>
> What he means is, "I didn't test this code before running it, and I am wrong."
>
> You cannot rebind __debug__.
>
>>>> __debug__ = False
>   File "<stdin>", line 1
> SyntaxError: can not assign to __debug__
>
>
> (That's Python 2.5 or better, and maybe even older than that.)

Andrew didn't assign directly to __debug__. He assigned to
sys.modules[__name__].__debug__, which is allowed prior to 2.7. Even
in later versions, as he pointed out, you can dynamically assign to
'__debug__' in a namespace dict.

For an expression containing __debug__, i.e. not simply an `if
__debug__` test, the compiler emits a LOAD_NAME operation (or
LOAD_GLOBAL in a function) to load the value on the stack and evaluate
the expression. Thus the builtins value may be shadowed by a local or
global value, or you can just modify builtins directly.

This inconsistency could be addressed by making __debug__ a static
symbol, which in CPython would always be determined by the value of
the C global variable Py_OptimizeFlag. In this case a code block would
reference a '__debug__' constant in its co_consts, defined at compile
time.

For example, this is the current inconsistent compile-time vs run-time
behavior, tested in 3.5:

    >>> import sys
    >>> sys.flags.optimize
    0
    >>> vars(sys.modules['builtins'])['__debug__'] = False

Compiling a simple `if __debug__` block depends on the value of Py_OptimizeFlag:

    >>> if __debug__: print('__debug__')
    ...
    __debug__

But evaluating an expression requires loading the dynamic value of __debug__:

    >>> if not __debug__: print('not __debug__')
    ...
    not __debug__

    >>> vars(sys.modules['builtins'])['__debug__'] = True
    >>> if not not __debug__: print('__debug__')
    ...
    __debug__

For `if __debug__` blocks, the compiler looks at the value of the C
global variable Py_OptimizeFlag:

    >>> import ctypes
    >>> Py_OptimizeFlag = ctypes.c_int.in_dll(
    ...     ctypes.pythonapi, 'Py_OptimizeFlag')

    >>> if __debug__: print('__debug__')
    ... else: print('not __debug__')
    ...
    __debug__

    >>> Py_OptimizeFlag.value = 1
    >>> if __debug__: print('__debug__')
    ... else: print('not __debug__')
    ...
    not __debug__

Off topic:
The text after the question ID in a Stack Overflow URL is optional, so
I removed it from the message title.


From info at egenix.com  Wed Nov 16 11:55:51 2016
From: info at egenix.com (eGenix Team: M.-A. Lemburg)
Date: Wed, 16 Nov 2016 17:55:51 +0100
Subject: ANN: eGenix mxODBC 3.3.6 - Python ODBC Database Interface
Message-ID: <582C8F97.1070307@egenix.com>

________________________________________________________________________

ANNOUNCING

                         eGenix.com mxODBC

                   Python ODBC Database Interface

                           Version 3.3.6


    mxODBC is our commercially supported Python extension providing
         ODBC database connectivity to Python applications
            on Windows, Mac OS X, Unix and BSD platforms
           with many advanced Python DB-API extensions and
                 full support of stored procedures


This announcement is also available on our web-site for online reading:
http://www.egenix.com/company/news/eGenix-mxODBC-3.3.6-GA.html

________________________________________________________________________

INTRODUCTION

mxODBC provides an easy-to-use, high-performance, reliable and robust
Python interface to ODBC compatible databases such as MS SQL Server,
Oracle Database, IBM DB2, Informix and Netezza, SAP Sybase ASE and
Sybase Anywhere, Teradata, MySQL, MariaDB, PostgreSQL, SAP MaxDB and
many more:

    http://www.egenix.com/products/python/mxODBC/

The "eGenix mxODBC - Python ODBC Database Interface" product is a
commercial extension to our open-source eGenix mx Base Distribution:

    http://www.egenix.com/products/python/mxBase/

________________________________________________________________________

NEWS

The 3.3.6 release of our mxODBC is a patch level release of our
popular Python ODBC Interface for Windows, Linux, Mac OS X and
FreeBSD. It includes these enhancements and fixes:

Features
--------

 * The mxODBC default *RowFactory helpers will no longer try to add
   column names which are not valid Python identifiers to the Row
   class code. Such columns are still available via index
   (e.g. row[0]) or named access (e.g. row['123']).

Bug Fixes
---------

 * IMPORTANT: Fixed a bug in context managers not properly detecting
   exceptions. This resulted exceptions getting silenced, transactions
   not getting committed and could lead to data corruption. Thanks to
   Jan Murre for the report.

For the full set of changes please check the mxODBC change log:

    http://www.egenix.com/products/python/mxODBC/changelog.html

________________________________________________________________________

FEATURES

mxODBC 3.3 was released on 2014-04-08. Please see the full
announcement for highlights of the 3.3 release:

    http://www.egenix.com/company/news/eGenix-mxODBC-3.3.0-GA.html

For the full set of features mxODBC has to offer, please see:

    http://www.egenix.com/products/python/mxODBC/#Features

________________________________________________________________________

EDITIONS

mxODBC is available in these two editions:

 * The Professional Edition, which gives full access to all mxODBC features.

 * The Product Development Edition, which allows including mxODBC in
   applications you develop.

For a complete overview of the available editions, please see the
product page:

    http://www.egenix.com/products/python/mxODBC/#mxODBCEditions

________________________________________________________________________

DOWNLOADS

The download archives and instructions for installing the package can
be found at:

    http://www.egenix.com/products/python/mxODBC/

In order to use the eGenix mxODBC package you will first need to
install the eGenix mx Base package:

    http://www.egenix.com/products/python/mxBase/

You can also simply use:

    pip install egenix-mxodbc

and then get evaluation licenses from our website to try mxODBC:

    http://www.egenix.com/products/python/mxODBC/#Evaluation

________________________________________________________________________

UPGRADING

Users are encouraged to upgrade to this latest mxODBC release to
benefit from the new features and updated ODBC driver support.

We have taken special care not to introduce backwards incompatible
changes, making the upgrade experience as smooth as possible.

Customers who have purchased mxODBC 3.3 licenses can continue to use
their licenses with this patch level release.

For upgrade purchases, we will give out 20% discount coupons going
from mxODBC 2.x to 3.3 and 50% coupons for upgrades from mxODBC 3.x to
3.3. Please contact the eGenix.com Sales Team with your existing
license serials for details for an upgrade discount coupon.

If you want to try the new release before purchase, you can request
30-day evaluation licenses by visiting our web-site

    http://www.egenix.com/products/python/mxODBC/#Evaluation

or writing to sales at egenix.com, stating your name (or the name of the
company) and the number of eval licenses that you need.

_______________________________________________________________________

SUPPORT

Commercial support for this product is available from eGenix.com.
Please see

    http://www.egenix.com/services/support/

for details about our support offerings.

_______________________________________________________________________

INFORMATION

About eGenix (http://www.egenix.com/):

    eGenix is a database focused software project, consulting and
    product company delivering expert services and professional
    quality products for companies, Python users and developers.

About Python (http://www.python.org/):

    Python is an object-oriented Open Source programming language
    which runs on all modern platforms. By integrating ease-of-use,
    clarity in coding, enterprise application connectivity and rapid
    application design, Python establishes an ideal programming
    platform for today's IT challenges.

Enjoy,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 16 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/



From ben+python at benfinney.id.au  Wed Nov 16 12:48:53 2016
From: ben+python at benfinney.id.au (Ben Finney)
Date: Thu, 17 Nov 2016 04:48:53 +1100
Subject: Farewell to Rob Collins
References: <582C3323.2000903@europython.eu>
Message-ID: <85eg2b5n96.fsf@benfinney.id.au>

"M.-A. Lemburg" <mal at europython.eu> writes:

> Rob was a true Pythonista from the heart. He will always be remembered
> for his humor, great spirit and kindness.

Robert and I had many conversations about the Bazaar version control
system, and I owe to him my passion for distributed version control.

When I needed to convince my workplace to try this strange new idea,
Robert kindly donated his time to visit our offices and give a
presentation on why distributed version control was better and why
Bazaar was a good choice for us.

He also inspired me to get serious about unit testing as a way to
increase confidence in a code base.

You touched many lives, and you are missed. Vale, Rob.

-- 
 \         ?What you have become is the price you paid to get what you |
  `\                                 used to want.? ?Mignon McLaughlin |
_o__)                                                                  |
Ben Finney



From rgaddi at highlandtechnology.invalid  Wed Nov 16 12:52:56 2016
From: rgaddi at highlandtechnology.invalid (Rob Gaddi)
Date: Wed, 16 Nov 2016 17:52:56 -0000 (UTC)
Subject: Access to the caller's globals, not your own
References: <582949b2$0$11109$c3e8da3@news.astraweb.com>
 <o0e4fp$tjb$1@dont-email.me> <582c1a73$0$11119$c3e8da3@news.astraweb.com>
Message-ID: <o0i6do$786$1@dont-email.me>

Steven D'Aprano wrote:

> On Tuesday 15 November 2016 15:55, Dan Sommers wrote:
>
>> On Mon, 14 Nov 2016 16:20:49 +1100, Steven D'Aprano wrote:
>> 
>>> import library
>>> SPAMIFY = False  # only affects this module, no other modules
>>> result = library.make_spam(99)
>> 
>> I must be missing something, because it seems too obvious:
> [snip]
>
> I wouldn't say "obvious" so much as "complex".
>
> A factory function or a class that holds state for a local make_spam() callable 
> would be a possible solution, but it significantly increases the complexity, 
> especially for the simple case:
>
> import library
> result = library.make_spam(arg)
>
>
> versus:
>
> import library
> make_spam = library.make_library()
> result = make_spam(arg)
>
> What a drag.
>
>

And there you have it; an entire extra line at import time.

It's the answer that's explicit.  It's versatile (you can create
multiple library instances with different defaults if that sort of thing
is really your jam), and it uses no tricky mechanisms that unskilled
programmers won't understand.

You can even create a default object in the main library with some
sensible defaults and bind out the methods as functions just to provide
a quick and easy answer for people who don't care.

  class Library:
    ...

  _defaultlib = Library()
  _defaultlib.rounding = NEAREST_EVEN
  make_spam = _defaultlib.make_spam

-- 
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order.  See above to fix.


From linuxmailinglistsemail at gmail.com  Wed Nov 16 14:02:31 2016
From: linuxmailinglistsemail at gmail.com (Sean Son)
Date: Wed, 16 Nov 2016 14:02:31 -0500
Subject: Farewell to Rob Collins
In-Reply-To: <85eg2b5n96.fsf@benfinney.id.au>
References: <582C3323.2000903@europython.eu> <85eg2b5n96.fsf@benfinney.id.au>
Message-ID: <CAOKpjz_RO6ZDnqsxHiqpQmfASbONmXUYQJCv5P3q_CzS+nr_3g@mail.gmail.com>

Rest in Peace Rob

On Wed, Nov 16, 2016 at 12:48 PM, Ben Finney <ben+python at benfinney.id.au>
wrote:

> "M.-A. Lemburg" <mal at europython.eu> writes:
>
> > Rob was a true Pythonista from the heart. He will always be remembered
> > for his humor, great spirit and kindness.
>
> Robert and I had many conversations about the Bazaar version control
> system, and I owe to him my passion for distributed version control.
>
> When I needed to convince my workplace to try this strange new idea,
> Robert kindly donated his time to visit our offices and give a
> presentation on why distributed version control was better and why
> Bazaar was a good choice for us.
>
> He also inspired me to get serious about unit testing as a way to
> increase confidence in a code base.
>
> You touched many lives, and you are missed. Vale, Rob.
>
> --
>  \         ?What you have become is the price you paid to get what you |
>   `\                                 used to want.? ?Mignon McLaughlin |
> _o__)                                                                  |
> Ben Finney
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From vmahajan at centerpointmedia.com  Wed Nov 16 15:01:17 2016
From: vmahajan at centerpointmedia.com (vmahajan at centerpointmedia.com)
Date: Wed, 16 Nov 2016 12:01:17 -0800 (PST)
Subject: Printing a generator returns "<generator object _as_iterable at
 0x7f0476373e10>", need to print its values
Message-ID: <d471f339-b558-4a22-be0d-aaad6f17ae70@googlegroups.com>

I am running Python2.7, wherein I am running the following price of code:

y = m.predict(input_fn=lambda:input_fn(df_predict), as_iterable=True)
print ('Predictions: {}'.format(str(y)))

The output of the following is ""<generator object _as_iterable at 0x7f0476373e10>"

However, the desired output must be in the format [1 0 0 1 1 0 0 1].

Can someone help me print the output in this format


From python at mrabarnett.plus.com  Wed Nov 16 15:14:56 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Wed, 16 Nov 2016 20:14:56 +0000
Subject: Printing a generator returns "<generator object _as_iterable at
 0x7f0476373e10>", need to print its values
In-Reply-To: <d471f339-b558-4a22-be0d-aaad6f17ae70@googlegroups.com>
References: <d471f339-b558-4a22-be0d-aaad6f17ae70@googlegroups.com>
Message-ID: <27fbda32-7d6e-1293-7fe9-c3b2e8b1b4fb@mrabarnett.plus.com>

On 2016-11-16 20:01, vmahajan at centerpointmedia.com wrote:
> I am running Python2.7, wherein I am running the following price of code:
>
> y = m.predict(input_fn=lambda:input_fn(df_predict), as_iterable=True)
> print ('Predictions: {}'.format(str(y)))
>
> The output of the following is ""<generator object _as_iterable at 0x7f0476373e10>"
>
> However, the desired output must be in the format [1 0 0 1 1 0 0 1].
>
> Can someone help me print the output in this format
>
Pass the generator object to 'list':

y = list(m.predict(input_fn=lambda:input_fn(df_predict), as_iterable=True))
print('Predictions: {}'.format(y))



From __peter__ at web.de  Wed Nov 16 15:25:18 2016
From: __peter__ at web.de (Peter Otten)
Date: Wed, 16 Nov 2016 21:25:18 +0100
Subject: Printing a generator returns "<generator object _as_iterable at
 0x7f0476373e10>", need to print its values
References: <d471f339-b558-4a22-be0d-aaad6f17ae70@googlegroups.com>
Message-ID: <o0ifbb$mc9$1@blaine.gmane.org>

vmahajan at centerpointmedia.com wrote:

> I am running Python2.7, wherein I am running the following price of code:
> 
> y = m.predict(input_fn=lambda:input_fn(df_predict), as_iterable=True)
> print ('Predictions: {}'.format(str(y)))
> 
> The output of the following is ""<generator object _as_iterable at
> 0x7f0476373e10>"
> 
> However, the desired output must be in the format [1 0 0 1 1 0 0 1].
> 
> Can someone help me print the output in this format

You give no context, but my suggestion would be to try and omit the 

as_iterable=True

part...




From wants at no.mail.local  Wed Nov 16 16:29:29 2016
From: wants at no.mail.local (andy)
Date: Wed, 16 Nov 2016 22:29:29 +0100
Subject: how to print variable few time?
References: <7b3d77cb-c42d-48bb-876f-ef36d9dc27f4@googlegroups.com>
 <d5251$582874b1$54817df3$25998@news1.surfino.com>
 <lf5polzflic.fsf@ling.helsinki.fi>
Message-ID: <82ab8$582ccfb9$54817d94$11167@news1.surfino.com>

Sun, 13 Nov 2016 17:27:23 +0200 wrote Jussi Piitulainen:


>>> word=raw_input()
>>> print word*3
>>> 
>>> 
>>> with this code im getting - wordwordword.
>>> what changes i need to make to get - word word word - instead?
>>> 
>>> thanks
>>
>> using python3.x:
>>
>> word=input()
>> print((word+' ')*2, end='')
>> print(word)
> 
> print(*[word]*3)

thanks for this starred expression - it took me one afternoon to 
understand ;-)

one should search the library and https://www.python.org/dev/peps/
pep-0448/ and try to

>>> import this

for clarification.


From ben+python at benfinney.id.au  Wed Nov 16 16:41:39 2016
From: ben+python at benfinney.id.au (Ben Finney)
Date: Thu, 17 Nov 2016 08:41:39 +1100
Subject: Farewell to Rob Collins
References: <582C3323.2000903@europython.eu> <85eg2b5n96.fsf@benfinney.id.au>
Message-ID: <858tsj5ch8.fsf@benfinney.id.au>

Ben Finney <ben+python at benfinney.id.au> writes:

> Robert and I had many conversations [?].

My apologies for the confusion! This is not the same person.

I did not know the Rob Collins described in the obituary
<URL:http://www.europython-society.org/post/153253946400/farewell-to-rob-collins>
so my comments were misguided.


Thank you, Rob Collins (of the UK, and of EuroPython fame), for your
contributions to free software.

-- 
 \       ?? one of the main causes of the fall of the Roman Empire was |
  `\        that, lacking zero, they had no way to indicate successful |
_o__)                  termination of their C programs.? ?Robert Firth |
Ben Finney



From jfong at ms4.hinet.net  Wed Nov 16 20:54:05 2016
From: jfong at ms4.hinet.net (jfong at ms4.hinet.net)
Date: Wed, 16 Nov 2016 17:54:05 -0800 (PST)
Subject: help on "from deen import *" vs. "import deen"
In-Reply-To: <582ba948$0$1589$c3e8da3$5496439d@news.astraweb.com>
References: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>
 <ca14fafb-2226-493e-ada5-e850a3c74f48@googlegroups.com>
 <e0478664-9610-4d83-8f58-583682d11fcd@googlegroups.com>
 <bb816fbf-1df7-522d-bab7-9f7a65deaef3@mrabarnett.plus.com>
 <mailman.147.1479180690.30068.python-list@python.org>
 <1a6f991d-53c1-4c79-b81b-e804b01be653@googlegroups.com>
 <144b0566-6738-bd37-cd1c-1585e70761d2@gmail.com>
 <9b6ad48d-ee65-bca6-6fa7-4176ddaae83e@lucidity.plus.com>
 <mailman.153.1479248175.30068.python-list@python.org>
 <582ba948$0$1589$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <5dcc9d11-a336-41f5-bed6-5e073f614bd0@googlegroups.com>

Steve D'Aprano at 2016/11/16 8:33:23AM wrote:
> `import foo` imports the module foo, that is all. (To be pedantic: it is
> *nominally* a module. By design, it could be any object at all.)
> 
> `from foo import *` imports all the visible public attributes of foo.
> 
> They do completely different things, equivalent to something similar to:
> 
> five = int('5')
> 
> 
> versus:
> 
> 
> _tmp = int('5')
> for name in dir(_tmp):
>     if not name.startswith('_'):
>         locals()[name] = getattr(_tmp, name)
> del _tmp

This is far beyond my comprehension:-(

--Jach



From jfong at ms4.hinet.net  Wed Nov 16 21:01:30 2016
From: jfong at ms4.hinet.net (jfong at ms4.hinet.net)
Date: Wed, 16 Nov 2016 18:01:30 -0800 (PST)
Subject: help on "from deen import *" vs. "import deen"
In-Reply-To: <mailman.154.1479266093.30068.python-list@python.org>
References: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>
 <ca14fafb-2226-493e-ada5-e850a3c74f48@googlegroups.com>
 <e0478664-9610-4d83-8f58-583682d11fcd@googlegroups.com>
 <bb816fbf-1df7-522d-bab7-9f7a65deaef3@mrabarnett.plus.com>
 <mailman.147.1479180690.30068.python-list@python.org>
 <1a6f991d-53c1-4c79-b81b-e804b01be653@googlegroups.com>
 <144b0566-6738-bd37-cd1c-1585e70761d2@gmail.com>
 <mailman.151.1479221023.30068.python-list@python.org>
 <80870bc5-5975-43e0-bf23-dccb1f628d6f@googlegroups.com>
 <5d48ae5f-17f1-fca7-1239-d22c2950363c@gmail.com>
 <mailman.154.1479266093.30068.python-list@python.org>
Message-ID: <c4c12dd3-7d2d-4a87-bdbc-0e752dbf5318@googlegroups.com>

Michael Torrie at 2016/11/16 11:15:11AM wrote:
> ... The globals object is a dictionary and is itself
> mutable.  But when we assign a new object to a particular dictionary
> key, it tosses out the old reference and makes the key now refer to the
> new object.  It does not do anything to the old object itself.

The last question: Is it possible, in the current Python version, to re-bind a global name in module "deen" after it was imported "from deen import *"?

> > That's one problem I was concerned. Human beings are very deeply
> > binding on words (names?). We think in words, talk in words,
> > communicate in words. Actually we are living in words. I really don't
> > like that when I said "John is a boy" and was told "No, John is a dog
> > now":-)
> 
> Not quite sure where you're going with that. I would think the idea of
> labels on objects would be fairly natural. I could stick a label that
> says "chair" on a chair, and then later move that label to the couch.
> Same label, different object.  Whether the label makes sense is up to you.

I mean it might be better to have two labels to label chair and couch separately, instead of one:-)

--Jach



From rosuav at gmail.com  Wed Nov 16 21:26:18 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 17 Nov 2016 13:26:18 +1100
Subject: help on "from deen import *" vs. "import deen"
In-Reply-To: <c4c12dd3-7d2d-4a87-bdbc-0e752dbf5318@googlegroups.com>
References: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>
 <ca14fafb-2226-493e-ada5-e850a3c74f48@googlegroups.com>
 <e0478664-9610-4d83-8f58-583682d11fcd@googlegroups.com>
 <bb816fbf-1df7-522d-bab7-9f7a65deaef3@mrabarnett.plus.com>
 <mailman.147.1479180690.30068.python-list@python.org>
 <1a6f991d-53c1-4c79-b81b-e804b01be653@googlegroups.com>
 <144b0566-6738-bd37-cd1c-1585e70761d2@gmail.com>
 <mailman.151.1479221023.30068.python-list@python.org>
 <80870bc5-5975-43e0-bf23-dccb1f628d6f@googlegroups.com>
 <5d48ae5f-17f1-fca7-1239-d22c2950363c@gmail.com>
 <mailman.154.1479266093.30068.python-list@python.org>
 <c4c12dd3-7d2d-4a87-bdbc-0e752dbf5318@googlegroups.com>
Message-ID: <CAPTjJmpjUVadTVQNeqc5Vqri_cK=Lq2GJMe1V__bHBxtyabPUA@mail.gmail.com>

On Thu, Nov 17, 2016 at 1:01 PM,  <jfong at ms4.hinet.net> wrote:
> Michael Torrie at 2016/11/16 11:15:11AM wrote:
>> ... The globals object is a dictionary and is itself
>> mutable.  But when we assign a new object to a particular dictionary
>> key, it tosses out the old reference and makes the key now refer to the
>> new object.  It does not do anything to the old object itself.
>
> The last question: Is it possible, in the current Python version, to re-bind a global name in module "deen" after it was imported "from deen import *"?

Yes.

from deen import *

...

import deen
deen.some_name = new_value

Voila :)

ChrisA


From orgnut at yahoo.com  Wed Nov 16 22:28:37 2016
From: orgnut at yahoo.com (Larry Hudson)
Date: Wed, 16 Nov 2016 19:28:37 -0800
Subject: A question about sprite rendering in game development
In-Reply-To: <0c4cb2a3-8678-40ad-b966-ab7c985b8875@googlegroups.com>
References: <746749d6-342d-43f2-ae48-8ea2a5118133@googlegroups.com>
 <0c4cb2a3-8678-40ad-b966-ab7c985b8875@googlegroups.com>
Message-ID: <3Jqdnclu2s57vrDFnZ2dnUU7-IHNnZ2d@giganews.com>

On 11/16/2016 12:16 AM, shadecelebi wrote:
> thanx a lot you guys. I'm slightly familiar with pygame from before so I'll make sure to utilize it. and no I don't have any of the characters yet as I've yet to start. I just wanted to know if I should keep learning python or if it would be trivial to try making my game a reality with this language. I'll need to learn a couple more things before I try to make a prototype. I believe the easiest route will be to simply rectangels of different colors for the early gameplay and to try and perfect the game mechanics before I move over to adding sprites and such. thank you both. I'll post updates in a different thread when I'm starting to see some kind of results
>

There's a book, "Making Games with Python & Pygame", available at 
http://inventwithpython.com/pygame/ you might find worth checking out.

On this site, there are two prominent links to "Buy It" and "Read it on-line".  But below these 
are three 'alternate' links to downloadable PDF and e-Reader versions as well as the source code 
(including the graphics used in the book).  The "Buy It" is for a hard-copy version.

-- 
      -=- Larry -=-


From steve+comp.lang.python at pearwood.info  Wed Nov 16 23:06:04 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Thu, 17 Nov 2016 15:06:04 +1100
Subject: help on "from deen import *" vs. "import deen"
References: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>
 <ca14fafb-2226-493e-ada5-e850a3c74f48@googlegroups.com>
 <e0478664-9610-4d83-8f58-583682d11fcd@googlegroups.com>
 <bb816fbf-1df7-522d-bab7-9f7a65deaef3@mrabarnett.plus.com>
 <mailman.147.1479180690.30068.python-list@python.org>
 <1a6f991d-53c1-4c79-b81b-e804b01be653@googlegroups.com>
 <144b0566-6738-bd37-cd1c-1585e70761d2@gmail.com>
 <9b6ad48d-ee65-bca6-6fa7-4176ddaae83e@lucidity.plus.com>
 <mailman.153.1479248175.30068.python-list@python.org>
 <582ba948$0$1589$c3e8da3$5496439d@news.astraweb.com>
 <5dcc9d11-a336-41f5-bed6-5e073f614bd0@googlegroups.com>
Message-ID: <582d2cae$0$1517$c3e8da3$5496439d@news.astraweb.com>

On Thursday 17 November 2016 12:54, jfong at ms4.hinet.net wrote:

> Steve D'Aprano at 2016/11/16 8:33:23AM wrote:
>> `import foo` imports the module foo, that is all. (To be pedantic: it is
>> *nominally* a module. By design, it could be any object at all.)
>> 
>> `from foo import *` imports all the visible public attributes of foo.
>> 
>> They do completely different things, equivalent to something similar to:
>> 
>> five = int('5')
>> 
>> 
>> versus:
>> 
>> 
>> _tmp = int('5')
>> for name in dir(_tmp):
>>     if not name.startswith('_'):
>>         locals()[name] = getattr(_tmp, name)
>> del _tmp
> 
> This is far beyond my comprehension:-(

That's okay, I was talking mostly to Eric.

You understand how this works?

five = int('5')


creates a new variable "five", and gives it the value 5.

py> five
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'five' is not defined
py> five = int("5")  # creates the new variable
py> five
5

That is similar to how import works: "import os" reads the "os.py" file, 
compiles it to a module object, creates the variable "os", and sets collections 
to that module object:

py> os
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'os' is not defined
py> import os
py> os
<module 'os' from '/usr/lib/python2.5/os.pyc'>



But that's not what "from os import *" does. It is more complicated. It looks 
inside the "os" module, extracts all the public functions, classes and global 
variables, and creates new variables in your own module for them.

The os module has close to 200 or more public functions and globals, so I'll 
pick a simpler example. Create this two line file:


# library.py
apple = 1
pear = 2



Now let's check that "import" works the way we expect. Inside the Python 
interactive interpreter, run this code:


py> library
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'library' is not defined
py> import library
py> library
<module 'library' from 'library.py'>


Do you get the same output?

Look inside the module:

py> print(library.apple)
1



Check that there's no "apple" variable, then import it:

py> apple
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'apple' is not defined
py> from library import apple
py> apple
1


Finally try this:

py> apple = 999
py> from library import *
py> apple
1
py> pear
2




Remember that the "apple" variable is a different variable from the 
"library.apple" variable. Think of:

# in the USA
president = "Barack Obama"

# in Russia
president = "Vladimir Putin"


Two variables, with the same name, but their values are independent:

usa.president = "Donald Trump"  # will happen soon
print(russia.president)  # still prints "Vladimir Putin"


The same happens when you are working inside the current active module:

president = "Xi Jinping"

This doesn't affect either usa.president or russia.president -- not even if we 
first take the value from the other module:


from russia import president
assert president == 'Vladimir Putin'  # this is true, for now
president = 'Francois Hollande'  # new president, here


Changing the president in the current module doesn't change the 
russia.president.



Now:

    from russia import president

is almost the same as:


    import russia as _tmp
    president = _tmp.president
    del _tmp



But this is more complicated:

    from russia import *


The star import has to work out all the visible public names, and create new 
variables for all of them. You don't need to understand how it does that, the 
details are not important. What is important is that the variables it creates 
are different from the variables living inside the russia module.




-- 
Steven
299792.458 km/s ? not just a good idea, it?s the law!



From steve+comp.lang.python at pearwood.info  Thu Nov 17 00:17:51 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Thu, 17 Nov 2016 16:17:51 +1100
Subject: Access to the caller's globals, not your own
References: <582949b2$0$11109$c3e8da3@news.astraweb.com>
 <o0e4fp$tjb$1@dont-email.me> <582c1a73$0$11119$c3e8da3@news.astraweb.com>
 <o0i6do$786$1@dont-email.me>
Message-ID: <582d3d81$0$1511$c3e8da3$5496439d@news.astraweb.com>

On Thursday 17 November 2016 04:52, Rob Gaddi wrote:

>> import library
>> result = library.make_spam(arg)
>>
>>
>> versus:
>>
>> import library
>> make_spam = library.make_library()
>> result = make_spam(arg)
>>
>> What a drag.
>>
>>
> 
> And there you have it; an entire extra line at import time.

Multiplied by every function that has a configurable default. Multiplied by 
every module that imports any of those functions, *whether or not* they change 
the default. If there are four functions, the user has to do this:

import library
make_spam = library.make_spam_factory()
make_eggs = library.make_eggs_factory()
make_cheese = library.make_cheese_factory()
make_toast = library.make_toast_factory()


before they can even begin to do what they actually want to do, which is make 
spam, eggs, cheese etc. And yes, there's likely to be four or more of these 
functions.

You can slightly work around this by offering pre-built functions that avoid 
needing to call the factory, but this is adding more complexity to what really 
isn't that complicated: a function with a user-configurable default setting.


> It's the answer that's explicit.

An explicit solution would be to offer no default setting at all and require 
the user to always provide the setting as an explicit argument to the function.

But despite the Zen of Python (which has always been intended as a humorous and 
intentionally self-contradictory way to think about the design of the language, 
not as a way to shut down discussion), explicit is not ALWAYS better than 
implicit. That's why we have default values, and why we don't require imports 
to be written like this:

    # explicitly giving the name to bind to is better
    import math as math


It's very unusual to require the caller of a library function to create the 
function first: factory functions and builders are a powerful technique, but 
they're more useful for the library, less so for the user of the library.

Conceptually, my library offers a simple function. The API of functions is 
well-known:

import library
result = library.function(arg)

The API for providing user-configurable default values is well-known and easily 
understood: you set a global variable (or even an environment variable):

DEFAULT = foo
result = library.function()

There's about half a century of programming practice and probably ten billion 
person-hours of collective experience behind this idea.

Whereas fancy tricks involving factory functions, builders and class-based 
solutions are *much* less well-known or understood. Outside of functional 
programming circles, the idea of calling a function to return a function is 
often considered mind-blowing voodoo.


> It's versatile (you can create
> multiple library instances with different defaults if that sort of thing
> is really your jam), 

Indeed: factory functions are great. But I'm saying that as the writer of the 
library, not the user of the library. Can you imagine expecting users to do 
this?

from math import trig
sin = trig.build('sine')
result = sin(0.1)

It might only be one extra line of code, but conceptually it is an order of 
magnitude more complex. The user has to comprehend factory functions and first-
class functions as values before they can even begin to understand this.


> and it uses no tricky mechanisms that unskilled
> programmers won't understand.

The internal details of how the implementation works is not important to the 
user. They only need to understand that the "scheme" parameter takes its value 
from the first found of:

- an explicit argument
- a global variable called "WHATEVER"
- the library default

which is pretty simple to understand and use.



-- 
Steven
299792.458 km/s ? not just a good idea, it?s the law!



From vek.m1234 at gmail.com  Thu Nov 17 00:40:03 2016
From: vek.m1234 at gmail.com (Veek M)
Date: Thu, 17 Nov 2016 11:10:03 +0530
Subject: What exactly is a python variable?
Message-ID: <o0jfql$cn$1@dont-email.me>

In C:
int x = 10; 
results in storage being allocated and type and location are fixed for 
the life of the program. 

In Python,
x = 10

causes an object '10' to be created but how exactly is 'x' handled? 
Symbol Table lookup at compile time? Is every 'x' being substituted out 
of existence? Because type(x) gives 'int' so..


From dan at tombstonezero.net  Thu Nov 17 00:40:04 2016
From: dan at tombstonezero.net (Dan Sommers)
Date: Thu, 17 Nov 2016 05:40:04 -0000 (UTC)
Subject: Access to the caller's globals, not your own
References: <582949b2$0$11109$c3e8da3@news.astraweb.com>
 <o0e4fp$tjb$1@dont-email.me> <582c1a73$0$11119$c3e8da3@news.astraweb.com>
 <o0i6do$786$1@dont-email.me>
 <582d3d81$0$1511$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o0jfrj$oa9$1@dont-email.me>

On Thu, 17 Nov 2016 16:17:51 +1100, Steven D'Aprano wrote:

> ... factory functions are great. But I'm saying that as the writer of
> the library, not the user of the library. Can you imagine expecting
> users to do this?

> from math import trig
> sin = trig.build('sine')
> result = sin(0.1)

No, but I could expect users to do this:

    import math # trig functions take radians by default
    math = math.degree_math # but I like degrees

Or to choose from one of these:

    import math # trig functions take radians by default
    import math.degree_math as math # but I like degrees

Or to choose from one of these:

    import math.radian_math as math # use the radians version
    import math.degree_math as math # use the degrees version

The complexity is taken on once, by the library author, who (presumably)
understands the complexity better than the users do.

> It might only be one extra line of code, but conceptually it is an
> order of magnitude more complex. The user has to comprehend factory
> functions and first-class functions as values before they can even
> begin to understand this.

It is only one line of code, but only the library author has to deal
with the rest of that.

> The internal details of how the implementation works is not important
> to the user ...

On that we all agree.  :-)

> ... They only need to understand that the "scheme" parameter takes its
> value from the first found of:
> 
> - an explicit argument
> - a global variable called "WHATEVER"
> - the library default
> 
> which is pretty simple to understand and use.

Hey, wait a minute:  you've slipped another option into the mix because
you originally wanted a separate "global" setting for each module that
imports your library.  With a global global rather than a module global,
you might get something like this:

    import library # default is HAMMIFY
    library.default = library.SPAMMIFY # but I like SPAMMIFY instead

(At which point, I see your point about wanting the library to reach
"out" to some sort of global space rather than the user reaching "in" to
the library space after the fact.)


From rosuav at gmail.com  Thu Nov 17 00:48:18 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 17 Nov 2016 16:48:18 +1100
Subject: What exactly is a python variable?
In-Reply-To: <o0jfql$cn$1@dont-email.me>
References: <o0jfql$cn$1@dont-email.me>
Message-ID: <CAPTjJmrgYOcPyP9GjH81UpnMiWs0PQHBeOsObFEg75FFnJ7uGA@mail.gmail.com>

On Thu, Nov 17, 2016 at 4:40 PM, Veek M <vek.m1234 at gmail.com> wrote:
> In C:
> int x = 10;
> results in storage being allocated and type and location are fixed for
> the life of the program.
>
> In Python,
> x = 10
>
> causes an object '10' to be created but how exactly is 'x' handled?
> Symbol Table lookup at compile time? Is every 'x' being substituted out
> of existence? Because type(x) gives 'int' so..

Here:

http://nedbatchelder.com/text/names1.html

Enjoy!

ChrisA


From jfong at ms4.hinet.net  Thu Nov 17 01:48:10 2016
From: jfong at ms4.hinet.net (jfong at ms4.hinet.net)
Date: Wed, 16 Nov 2016 22:48:10 -0800 (PST)
Subject: help on "from deen import *" vs. "import deen"
In-Reply-To: <582d2cae$0$1517$c3e8da3$5496439d@news.astraweb.com>
References: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>
 <ca14fafb-2226-493e-ada5-e850a3c74f48@googlegroups.com>
 <e0478664-9610-4d83-8f58-583682d11fcd@googlegroups.com>
 <bb816fbf-1df7-522d-bab7-9f7a65deaef3@mrabarnett.plus.com>
 <mailman.147.1479180690.30068.python-list@python.org>
 <1a6f991d-53c1-4c79-b81b-e804b01be653@googlegroups.com>
 <144b0566-6738-bd37-cd1c-1585e70761d2@gmail.com>
 <9b6ad48d-ee65-bca6-6fa7-4176ddaae83e@lucidity.plus.com>
 <mailman.153.1479248175.30068.python-list@python.org>
 <582ba948$0$1589$c3e8da3$5496439d@news.astraweb.com>
 <5dcc9d11-a336-41f5-bed6-5e073f614bd0@googlegroups.com>
 <582d2cae$0$1517$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <f1ae52cf-6582-43ac-b65b-c7bad2b35f15@googlegroups.com>

Steven D'Aprano at 2016/11/17 12:06:19PM wrote:
> You understand how this works?

Yes, thank you for your detail explanation.

>     import russia as _tmp
>     president = _tmp.president
>     del _tmp

This one I can understand. But the previous one

>>_tmp = int('5')
>>for name in dir(_tmp):
>>    if not name.startswith('_'):
>>        locals()[name] = getattr(_tmp, name)
>>del _tmp

which I am still on scratching my head.

Now the question moves from "how" to "why":

Why "del _tmp" at the last step? The only reason I can thought of is "information hiding", but from whom? A global variable has its reason to be as a global. It may need to be modified later to influence others behavior. Using delete to hide the name seems unnecessary and redundant. If someone really want, he can follow the solution Chris had provided in his reply.

>>from deen import *
>>...
>>import deen
>>deen.some_name = new_value

A little strange, but effective:-)

--Jach



From steve+comp.lang.python at pearwood.info  Thu Nov 17 02:35:26 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Thu, 17 Nov 2016 18:35:26 +1100
Subject: __debug__ http://stackoverflow.com/questions/15305688
References: <CACL+1asTO3yF-y7E5JLUFfMrJ-NKC=ZCWt15AM6y0EKo4Z3MyA@mail.gmail.com>
 <mailman.164.1479309813.30068.python-list@python.org>
Message-ID: <582d5dc1$0$1522$c3e8da3$5496439d@news.astraweb.com>

On Thursday 17 November 2016 02:22, eryk sun wrote:

> On Wed, Nov 16, 2016 at 8:39 AM, Steven D'Aprano
> <steve+comp.lang.python at pearwood.info> wrote:
>> On Wednesday 16 November 2016 16:21, Veek M wrote:
>>
>>> Trying to make sense of that article. My understanding of debug was
>>> simple:
>>> 1. __debug__ is always True, unless -O or -OO
>>> 2. 'if' is optimized out when True and the expr is inlined.
>>>
>>> So what does he mean by:
>>>
>>> 1. 'If you rebind __debug__, it can cause symptoms'
>>
>> What he means is, "I didn't test this code before running it, and I am
>> wrong."
>>
>> You cannot rebind __debug__.
>>
>>>>> __debug__ = False
>>   File "<stdin>", line 1
>> SyntaxError: can not assign to __debug__
>>
>>
>> (That's Python 2.5 or better, and maybe even older than that.)
> 
> Andrew didn't assign directly to __debug__. He assigned to
> sys.modules[__name__].__debug__, which is allowed prior to 2.7. Even
> in later versions, as he pointed out, you can dynamically assign to
> '__debug__' in a namespace dict.


That's a fascinating loophole.

In any case, that's not something people are likely to stumble onto by 
accident, and if you're intentionally writing to a reserved name in a non-
standard way, whatever side-effects you trip over are your own fault.


>     >>> if not __debug__: print('not __debug__')
>     ...
>     not __debug__

That should be a candidate for keyhole optimization, same as `if __debug__`.



-- 
Steven
299792.458 km/s ? not just a good idea, it?s the law!



From steve+comp.lang.python at pearwood.info  Thu Nov 17 03:03:50 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Thu, 17 Nov 2016 19:03:50 +1100
Subject: help on "from deen import *" vs. "import deen"
References: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>
 <ca14fafb-2226-493e-ada5-e850a3c74f48@googlegroups.com>
 <e0478664-9610-4d83-8f58-583682d11fcd@googlegroups.com>
 <bb816fbf-1df7-522d-bab7-9f7a65deaef3@mrabarnett.plus.com>
 <mailman.147.1479180690.30068.python-list@python.org>
 <1a6f991d-53c1-4c79-b81b-e804b01be653@googlegroups.com>
 <144b0566-6738-bd37-cd1c-1585e70761d2@gmail.com>
 <9b6ad48d-ee65-bca6-6fa7-4176ddaae83e@lucidity.plus.com>
 <mailman.153.1479248175.30068.python-list@python.org>
 <582ba948$0$1589$c3e8da3$5496439d@news.astraweb.com>
 <5dcc9d11-a336-41f5-bed6-5e073f614bd0@googlegroups.com>
 <582d2cae$0$1517$c3e8da3$5496439d@news.astraweb.com>
 <f1ae52cf-6582-43ac-b65b-c7bad2b35f15@googlegroups.com>
Message-ID: <582d6467$0$1505$c3e8da3$5496439d@news.astraweb.com>

On Thursday 17 November 2016 17:48, jfong at ms4.hinet.net wrote:

> Steven D'Aprano at 2016/11/17 12:06:19PM wrote:
>> You understand how this works?
> 
> Yes, thank you for your detail explanation.
> 
>>     import russia as _tmp
>>     president = _tmp.president
>>     del _tmp
> 
> This one I can understand. But the previous one
> 
>>>_tmp = int('5')
>>>for name in dir(_tmp):
>>>    if not name.startswith('_'):
>>>        locals()[name] = getattr(_tmp, name)
>>>del _tmp
> 
> which I am still on scratching my head.


dir(obj) returns a list of all the attributes of obj:

py> dir(5)
['__abs__', '__add__', '__and__', '__class__', '__cmp__', '__coerce__', 
'__delattr__', '__div__', '__divmod__', '__doc__', '__float__', '__floordiv__', 
'__format__', '__getattribute__', '__getnewargs__', '__hash__', '__hex__', 
'__index__', '__init__', '__int__', '__invert__', '__long__', '__lshift__', 
'__mod__', '__mul__', '__neg__', '__new__', '__nonzero__', '__oct__', '__or__', 
'__pos__', '__pow__', '__radd__', '__rand__', '__rdiv__', '__rdivmod__', 
'__reduce__', '__reduce_ex__', '__repr__', '__rfloordiv__', '__rlshift__', 
'__rmod__', '__rmul__', '__ror__', '__rpow__', '__rrshift__', '__rshift__', 
'__rsub__', '__rtruediv__', '__rxor__', '__setattr__', '__sizeof__', '__str__', 
'__sub__', '__subclasshook__', '__truediv__', '__trunc__', '__xor__', 
'bit_length', 'conjugate', 'denominator', 'imag', 'numerator', 'real']


Most of those are __ dunder __ methods reserved by Python. Don't touch them.

So if you look at each name in turn, pick out the ones that DON'T begin with an 
underscore:

['bit_length', 'conjugate', 'denominator', 'imag', 'numerator', 'real']


Then use getattr(obj, name) to return the value of that attribute, which may be 
a method:

py> getattr(5, 'bit_length')
<built-in method bit_length of int object at 0x86ae2b0>


and then save it in the local namespace:

py> x = -1
py> locals()['x'] = 999
py> x
999


(But watch out: assignment using locals() doesn't always work, for technical 
reasons it may not work inside a function.)


> Now the question moves from "how" to "why":
> 
> Why "del _tmp" at the last step? 

Technically, there is no "del _tmp" because there is no _tmp created first. The 
import command doesn't create the _tmp variable in the first place, so there's 
no need to delete it.

But there's no easy way to do that from pure Python code, so the easiest way is 
to use a temporary variable and then delete it afterwards.


> The only reason I can thought of is
> "information hiding", but from whom? A global variable has its reason to be
> as a global. It may need to be modified later to influence others behavior.
> Using delete to hide the name seems unnecessary and redundant. 

This has very little to do with global variables. This is about importing 
names. If you want to import the module as a whole, then use

    import module

but if you want to import attributes of the module, then use:

    from module import foo, bar, baz

If you want both, do both.



The most important thing you should learn from this thread is:

- avoid using "from module import *" as it is usually more trouble 
  than it is worth.


It is confusing and leads to more problems than it solves. If Python was being 
invented now, rather than 20 years ago, I would expect that there would be no 
"import *" in the language.




-- 
Steven
299792.458 km/s ? not just a good idea, it?s the law!



From steve+comp.lang.python at pearwood.info  Thu Nov 17 03:09:29 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Thu, 17 Nov 2016 19:09:29 +1100
Subject: What exactly is a python variable?
References: <o0jfql$cn$1@dont-email.me>
Message-ID: <582d65b9$0$1505$c3e8da3$5496439d@news.astraweb.com>

On Thursday 17 November 2016 16:40, Veek M wrote:

> In C:
> int x = 10;
> results in storage being allocated and type and location are fixed for
> the life of the program.
> 
> In Python,
> x = 10
> 
> causes an object '10' to be created but how exactly is 'x' handled?
> Symbol Table lookup at compile time? 

No. Symbol table lookup at run time.

For functions, you can access a copy of the symbol table with:

locals()

You can access the actual global symbol table (not a copy) with:

globals()


Every module has its own independent global symbol table.



-- 
Steven
299792.458 km/s ? not just a good idea, it?s the law!



From chris at simplistix.co.uk  Thu Nov 17 06:37:11 2016
From: chris at simplistix.co.uk (Chris Withers)
Date: Thu, 17 Nov 2016 11:37:11 +0000
Subject: mush 2.4.0 released! - Type-based dependency injection for scripts
Message-ID: <0c0bac10-a5b3-1b2e-fde5-7fc488fbe6a2@simplistix.co.uk>

Hi All,

I'm very happy to announce the a new release of Mush, a light weight 
dependency injection framework aimed at enabling the easy testing and 
re-use of chunks of code that make up scripts.

This release includes:

- Add support for cloning depending on what label was used to add
   callables.

- Add Runner.add_label() helper to just add a label at the end of the
   runner.

- Document and flesh out Plugs.

- Switch to full Semantic Versioning.

For a worked example of how to use Mush to reduce the copy'n'paste in 
your scripts, please see here:

http://mush.readthedocs.io/en/latest/examples.html

Full docs are here:

http://mush.readthedocs.io/en/latest/index.html

Downloads are here:

https://pypi.python.org/pypi/mush

Compatible with Python 2.7, 3.3+ on Linux, Mac OS X and Windows.

Any problems, please give me a shout on the simplistix at googlegroups.com 
list!

cheers,

Chris


From bc at freeuk.com  Thu Nov 17 06:37:36 2016
From: bc at freeuk.com (BartC)
Date: Thu, 17 Nov 2016 11:37:36 +0000
Subject: What exactly is a python variable?
In-Reply-To: <o0jfql$cn$1@dont-email.me>
References: <o0jfql$cn$1@dont-email.me>
Message-ID: <o0k4p2$r0e$1@dont-email.me>

On 17/11/2016 05:40, Veek M wrote:
> In C:
> int x = 10;
> results in storage being allocated and type and location are fixed for
> the life of the program.
>
> In Python,
> x = 10
>
> causes an object '10' to be created but how exactly is 'x' handled?
> Symbol Table lookup at compile time? Is every 'x' being substituted out
> of existence? Because type(x) gives 'int' so..

Try:

  import dis

  def fn():
  |   global x
  |   x=10

  dis.dis(fn)

(I don't know how to disassemble code outside a function, not from 
inside the same program. Outside it might be: 'python -m dis file.py')

This might show stuff like:

               0 LOAD_CONST               1 (10)
               3 STORE_GLOBAL             0 (x)

So already giving a better idea of what might be going on compared with 
'x=10' which is practically the same in any language.

(I'm guessing "x" is looked up at this point, created if it doesn't 
exist, and associated with the value 'integer 10'.)

-- 
Bartc


From stanleydasilva93 at gmail.com  Thu Nov 17 06:55:30 2016
From: stanleydasilva93 at gmail.com (stanleydasilva93 at gmail.com)
Date: Thu, 17 Nov 2016 03:55:30 -0800 (PST)
Subject: Error in webscraping problem
Message-ID: <1d10586d-e517-462c-8397-b18c56fbd48c@googlegroups.com>

I am trying to solve the following problem.  Two numbers appear on a website. The user has to enter the gcd (greatest common divisor) and hit the 
submit button.  The catch is that the time limit is far too slow for
any human processes -- it must be fully automated. The numbers change
each time the user attempts the problem.

Unfortunately, I can't release the name of the website because of 
corporate confidentiality but I'm hoping someone may have some clues
as to what I'm doing wrong.  The code is below.  FinalResults.html
gives me a "Wrong Answer" message.

As a check, I kept a record of the results for one pair of numbers.
I called the webpage corresponding to a fixed pair "testRequest.txt"
The code:  

 myfile = open("testRequest.txt" , "r")
 page = myfile.read()

is used to check that the first number, second number, and solution, are
all as they should be.

I would be very grateful for any help or advice on the next steps.
Part of the problem is in debugging.

Thanks,
Stanley.


import sys
sys.path.append('C:/Users/silviadaniel/Anaconda3/Lib/site-packages')
import mechanize
import requests
import fractions
url = "http://someWebsite.com/test"
br = mechanize.Browser()
br.open(url)
br.select_form(nr = 0) 
# There is only one form involved so  this is probably ok

print br.form
# reads <POST http://website.com/submit application/x-www-form-urlencoded
# <TextControl(divisor=)>
# <SubmitButtonControl(<None>=) (readonly)>>

data = requests.get(url)
page = data.text
begTag = "<strong>"
endTag = "</strong>"
firstIndex = page.find(begTag)
secondIndex = page.find(endTag)
shift = len(begTag)
posNumber = firstIndex + shift
firstNumber = int(page[posNumber:secondIndex])
firstIndex = page.find(begTag, firstIndex + 1)
secondIndex = page.find(endTag, secondIndex + 1)
posNumber = firstIndex + shift
secondNumber = int(page[posNumber:secondIndex])
solution = str(fractions.gcd(firstNumber, secondNumber))
br["divisor"] = solution
print firstNumber # Looks sensible -- first number probably correct
print secondNumber # Looks sensible -- also checked
print solution # Is indeed the correct gcd
res = br.submit()

content = res.read()
with open("FinalResults.html", "w") as f:
    f.write(content)
# Unfortunately, I examine this file to find "Wrong Answer"


From steve+python at pearwood.info  Thu Nov 17 07:20:34 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Thu, 17 Nov 2016 23:20:34 +1100
Subject: What exactly is a python variable?
References: <o0jfql$cn$1@dont-email.me> <o0k4p2$r0e$1@dont-email.me>
Message-ID: <582da094$0$1590$c3e8da3$5496439d@news.astraweb.com>

On Thu, 17 Nov 2016 10:37 pm, BartC wrote:

> Try:
> 
>   import dis
> 
>   def fn():
>   |   global x
>   |   x=10
> 
>   dis.dis(fn)
> 
> (I don't know how to disassemble code outside a function, not from
> inside the same program. Outside it might be: 'python -m dis file.py')


You can use the byte-code compiler to compile the code first:


For an expression, you can use:

code = compile("x + 1", "", "eval")


The middle argument, shown here as an empty string "", is used for an
optional string identifying the source of the code. E.g. a file name.

The third argument, here shown as "eval", determines the compilation mode.
The eval() function can only evaluate a single expression, so the name of
the mode is the same.

For a single statement, as seen by the interactive interpreter, use:

code = compile("result = x + 1", "", "single")


For multiple statements (as in a module), or a single statement *not* in the
interactive interpreter, use:

code = compile("result = x + 1", "", "exec")

code = compile("""
x = 999
result = x + 1
print(result)
""", "", "exec")

Then once you have your code object, you can disassemble it:

dis.dis(code)

or exec/eval it:

exec(code)
eval(code)  # only if the compilation mode was "eval"


In the most recent versions of Python, dis.dis() will also accept a string:

py> dis.dis('y = x + 1')
  1           0 LOAD_NAME                0 (x)
              3 LOAD_CONST               0 (1)
              6 BINARY_ADD
              7 STORE_NAME               1 (y)
             10 LOAD_CONST               1 (None)
             13 RETURN_VALUE





-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From thorsten at thorstenkampe.de  Thu Nov 17 07:24:54 2016
From: thorsten at thorstenkampe.de (Thorsten Kampe)
Date: Thu, 17 Nov 2016 13:24:54 +0100
Subject: How to test for type or instance of dict_values?
Message-ID: <o0k7j9$29r$1@blaine.gmane.org>

How can I test for type or instance of dictviews like dict_values?

`isinstance({}.values, dict_values)` gives
`NameError: name 'dict_values' is not defined`

"""
>>> type({}.values())
<class 'dict_values'>
"""

Thorsten




From __peter__ at web.de  Thu Nov 17 07:38:26 2016
From: __peter__ at web.de (Peter Otten)
Date: Thu, 17 Nov 2016 13:38:26 +0100
Subject: How to test for type or instance of dict_values?
References: <o0k7j9$29r$1@blaine.gmane.org>
Message-ID: <o0k8c1$g3o$1@blaine.gmane.org>

Thorsten Kampe wrote:

> How can I test for type or instance of dictviews like dict_values?

Why do you want to?
 
> `isinstance({}.values, dict_values)` gives
> `NameError: name 'dict_values' is not defined`

You can "fix" this with

>>> dict_values = type({}.values())

or, depending on the use case, use another test:

>>> isinstance({}.values(), collections.abc.ValuesView)
True

> """
>>>> type({}.values())
> <class 'dict_values'>
> """
> 
> Thorsten




From bc at freeuk.com  Thu Nov 17 08:19:27 2016
From: bc at freeuk.com (BartC)
Date: Thu, 17 Nov 2016 13:19:27 +0000
Subject: What exactly is a python variable?
In-Reply-To: <582da094$0$1590$c3e8da3$5496439d@news.astraweb.com>
References: <o0jfql$cn$1@dont-email.me> <o0k4p2$r0e$1@dont-email.me>
 <582da094$0$1590$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o0kao0$f04$1@dont-email.me>

On 17/11/2016 12:20, Steve D'Aprano wrote:
> On Thu, 17 Nov 2016 10:37 pm, BartC wrote:

>> (I don't know how to disassemble code outside a function, not from
>> inside the same program. Outside it might be: 'python -m dis file.py')

> In the most recent versions of Python, dis.dis() will also accept a string:
>
> py> dis.dis('y = x + 1')
>   1           0 LOAD_NAME                0 (x)
>               3 LOAD_CONST               0 (1)
>               6 BINARY_ADD
>               7 STORE_NAME               1 (y)
>              10 LOAD_CONST               1 (None)
>              13 RETURN_VALUE


Py2 gave me (for "y=x+1"):

           0 SETUP_EXCEPT    30781 (to 30784)
           3 STORE_SLICE+3
           4 <49>

Py3.4 works as you say but after that result I was disinclined to take 
it further!

-- 
Bartc


From python at mrabarnett.plus.com  Thu Nov 17 08:55:43 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Thu, 17 Nov 2016 13:55:43 +0000
Subject: Access to the caller's globals, not your own
In-Reply-To: <o0jfrj$oa9$1@dont-email.me>
References: <582949b2$0$11109$c3e8da3@news.astraweb.com>
 <o0e4fp$tjb$1@dont-email.me> <582c1a73$0$11119$c3e8da3@news.astraweb.com>
 <o0i6do$786$1@dont-email.me>
 <582d3d81$0$1511$c3e8da3$5496439d@news.astraweb.com>
 <o0jfrj$oa9$1@dont-email.me>
Message-ID: <83db017b-23b3-9b65-9fa3-21b809ffe0f2@mrabarnett.plus.com>

On 2016-11-17 05:40, Dan Sommers wrote:
> On Thu, 17 Nov 2016 16:17:51 +1100, Steven D'Aprano wrote:
>
>> ... factory functions are great. But I'm saying that as the writer of
>> the library, not the user of the library. Can you imagine expecting
>> users to do this?
>
>> from math import trig
>> sin = trig.build('sine')
>> result = sin(0.1)
>
> No, but I could expect users to do this:
>
>     import math # trig functions take radians by default
>     math = math.degree_math # but I like degrees
>
> Or to choose from one of these:
>
>     import math # trig functions take radians by default
>     import math.degree_math as math # but I like degrees
>
> Or to choose from one of these:
>
>     import math.radian_math as math # use the radians version
>     import math.degree_math as math # use the degrees version
>
> The complexity is taken on once, by the library author, who (presumably)
> understands the complexity better than the users do.
>
[snip]
I wonder if it would be possible to make modules callable. You would do 
this by adding a __call__ function, and this function could be a factory 
function:

     import math
     degree_math = math('degree')

This would be equivalent to:

     import math
     degree_math = math.__call__('degree')

Also, a possible syntax extension:

     import math('degree') as degree_math

This would be equivalent to:

     import math as __temp__
     degree_math = __temp__.__call__('degree')
     del __temp__

If you wrote:

     import math('degree')

this would be equivalent to:

     import math as __temp__
     math = __temp__.__call__('degree')
     del __temp__



From arunsocs at gmail.com  Thu Nov 17 09:27:49 2016
From: arunsocs at gmail.com (arunsocs at gmail.com)
Date: Thu, 17 Nov 2016 06:27:49 -0800 (PST)
Subject: Overlapping co-ordiantes of rectangles fail to print in python
Message-ID: <1a9f480d-7e21-4559-aa38-a18899ca0b22@googlegroups.com>

 I am working with following code in which I am trying to output co ordinates of overlapping rectangles.. However the code fails to output the co ordinates. I am customizing the following code This is the input

1.6 1.2 7.9 3.1
1.2 1.6 3.4 7.2
2.6 11.6 6.8 14.0
9.6 1.2 11.4 7.5
9.6 1.7 14.1 2.8 

This is the code:

from __future__ import division
from collections import namedtuple
from itertools import combinations

Point = namedtuple( 'Point', ['x','y'] )

class Rectangle:
    def __init__( self, coordinates ):
        self.min = Point( coordinates[0], coordinates[1] )
        self.max = Point( coordinates[2], coordinates[3] )

    @property
    def center( self ):
        return Point( (self.max.x + self.min.x)/2, (self.max.y + self.min.y)/2 )

    @property
    def area( self ):
        return (self.max.x - self.min.x) * (self.max.y - self.min.y)

    def intersect( self, otherRect ):
        x_vals = sorted([self.max.x, self.min.x, otherRect.max.x, otherRect.min.x])
        y_vals = sorted([self.max.y, self.min.y, otherRect.max.y, otherRect.min.y])

        possibleIntersections = []
        intersections = []

        for i in range(3):
            for j in range(3):
                possibleIntersections.append( Rectangle([x_vals[i], y_vals[j], x_vals[i+1], y_vals[j+1]]) )

        for r in possibleIntersections:
            if self.contains( r.center ) and otherRect.contains( r.center ) and r.area > 0:
                intersections.append( r )

        return intersections

    def contains( self, point ):
        return self.min.x <= point.x and point.x <= self.max.x and self.min.y <= point.y and point.y <= self.max.y

    def __repr__( self ):
        return '[{0},{1}]'.format( self.min, self.max )

def readInputconvert( filename ):
    rects = []
    with open(filename,'r') as f:
        count = int(f.readline().rstrip());

        for _ in range( count ):
            rects.append( Rectangle( map( float, f.readline().rstrip().split(' ') ) ) )

    return rects

rectangles = readInputconvert( 'input.txt' ) # read input

sign = -1
area = sum( map( lambda x: x.area, rectangles) )

for i in range(2,len(rectangles)+1):
    for rects in combinations( rectangles, i ):
        intersections = [rects[0]]
        rects = rects[1:]
        for rectangle in rects:
            newintersections = []
            for otherR in intersections:
                newintersections.extend( rectangle.intersect(otherR) )

            intersections = newintersections
            print intersections

        #intersectingArea = sum( map( lambda x: x.area, intersections ) )
        #rea = area + (sign * intersectingArea)

    sign = sign*-1

Where I need to change the code to output all overlapping rectangles and its co ordinates?


From rosuav at gmail.com  Thu Nov 17 09:46:13 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 18 Nov 2016 01:46:13 +1100
Subject: Error in webscraping problem
In-Reply-To: <1d10586d-e517-462c-8397-b18c56fbd48c@googlegroups.com>
References: <1d10586d-e517-462c-8397-b18c56fbd48c@googlegroups.com>
Message-ID: <CAPTjJmqBxeHZFmA6DrejaVbMSBSsP-YgN_V1T_u_jGwaOsGafA@mail.gmail.com>

On Thu, Nov 17, 2016 at 10:55 python-ideas
<python-ideas at python.org>PM,  <stanleydasilva93 at gmail.com> wrote:
> I am trying to solve the following problem.  Two numbers appear on a website. The user has to enter the gcd (greatest common divisor) and hit the
> submit button.  The catch is that the time limit is far too slow for
> any human processes -- it must be fully automated. The numbers change
> each time the user attempts the problem.
>
> Unfortunately, I can't release the name of the website because of
> corporate confidentiality but I'm hoping someone may have some clues
> as to what I'm doing wrong.  The code is below.  FinalResults.html
> gives me a "Wrong Answer" message.

It's a programming challenge. Have fun with it! :)

ChrisA


From thorsten at thorstenkampe.de  Thu Nov 17 09:57:25 2016
From: thorsten at thorstenkampe.de (Thorsten Kampe)
Date: Thu, 17 Nov 2016 15:57:25 +0100
Subject: How to test for type or instance of dict_values?
References: <o0k7j9$29r$1@blaine.gmane.org> <o0k8c1$g3o$1@blaine.gmane.org>
Message-ID: <o0kgh9$pd6$1@blaine.gmane.org>

* Peter Otten (Thu, 17 Nov 2016 13:38:26 +0100)
> 
> Thorsten Kampe wrote:
> 
> > How can I test for type or instance of dictviews like dict_values?
> 
> Why do you want to?

Thanks, for the `collections.abc.ValuesView` tip.

The code in question is part of an attempt to get the dimensions of 
multi-dimensional lists, the `isinstance` is there in order to 
exclude strings.

"""
def dim(seq):
    dimension = []
    while isinstance(seq, (list, tuple)):
        dimension.append(len(seq))
        try:
            seq = seq[0]
        except IndexError:  # sequence is empty
            break
    return dimension
"""

Thorsten



From torriem at gmail.com  Thu Nov 17 10:38:08 2016
From: torriem at gmail.com (Michael Torrie)
Date: Thu, 17 Nov 2016 08:38:08 -0700
Subject: help on "from deen import *" vs. "import deen"
In-Reply-To: <c4c12dd3-7d2d-4a87-bdbc-0e752dbf5318@googlegroups.com>
References: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>
 <ca14fafb-2226-493e-ada5-e850a3c74f48@googlegroups.com>
 <e0478664-9610-4d83-8f58-583682d11fcd@googlegroups.com>
 <bb816fbf-1df7-522d-bab7-9f7a65deaef3@mrabarnett.plus.com>
 <mailman.147.1479180690.30068.python-list@python.org>
 <1a6f991d-53c1-4c79-b81b-e804b01be653@googlegroups.com>
 <144b0566-6738-bd37-cd1c-1585e70761d2@gmail.com>
 <mailman.151.1479221023.30068.python-list@python.org>
 <80870bc5-5975-43e0-bf23-dccb1f628d6f@googlegroups.com>
 <5d48ae5f-17f1-fca7-1239-d22c2950363c@gmail.com>
 <mailman.154.1479266093.30068.python-list@python.org>
 <c4c12dd3-7d2d-4a87-bdbc-0e752dbf5318@googlegroups.com>
Message-ID: <ababec33-ad77-60f4-367f-aa89aad22230@gmail.com>

On 11/16/2016 07:01 PM, jfong at ms4.hinet.net wrote:
> Michael Torrie at 2016/11/16 11:15:11AM wrote:
>> ... The globals object is a dictionary and is itself mutable.  But
>> when we assign a new object to a particular dictionary key, it
>> tosses out the old reference and makes the key now refer to the new
>> object.  It does not do anything to the old object itself.
> 
> The last question: Is it possible, in the current Python version, to
> re-bind a global name in module "deen" after it was imported "from
> deen import *"?

Sigh. If you understood what everyone has been explaining, you'd know
that the answer to that is no.  That's not the way variables in Python
work.  The only way to rebind deen.foo is to do as Chris suggested in
his reply and always refer to it with "deen.foo".

Once a variable is imported from a library it becomes a local variable
in your module (well a local global variable, since variables can only
be global to the module itself).  It starts out life referring to the
same object as the global attribute inside of deen. But once you assign
to the local name, it is rebound to a new object.  Do you understand that?


> I mean it might be better to have two labels to label chair and couch
> separately, instead of one:-)

Like I said, whether the names you use are appropriate is completely up
to you.  But this statement seems to imply you're still not getting it
and still thinking of variables as boxes like they are in other
languages, rather than labels that can only point to one thing at a
time.  Python's variables are different from other languages, but in an
understandable way.  Sit back and think about it, play with the python
command prompt. Run id() on a variable, assign something else to it, and
run id() on it again.  As Dennis said, understanding Python variables,
though not difficult, needs to be second nature to you or you'll be
fighting Python and hating the experience.


From amamaenko at gmail.com  Thu Nov 17 11:29:02 2016
From: amamaenko at gmail.com (Anton Mamaenko)
Date: Thu, 17 Nov 2016 19:29:02 +0300
Subject: What exactly is a python variable?
In-Reply-To: <o0kao0$f04$1@dont-email.me>
References: <o0jfql$cn$1@dont-email.me> <o0k4p2$r0e$1@dont-email.me>
 <582da094$0$1590$c3e8da3$5496439d@news.astraweb.com>
 <o0kao0$f04$1@dont-email.me>
Message-ID: <4E03EB8D-9A69-443C-9253-C03EF7CB9CBD@gmail.com>

Just a couple of days ago I was asking myself a similar question, and found this blog article: https://jeffknupp.com/blog/2013/02/14/drastically-improve-your-python-understanding-pythons-execution-model/

Clarified a lot of things to me. 

, Anton

> On 17 Nov 2016, at 16:19, BartC <bc at freeuk.com> wrote:
> 
>> On 17/11/2016 12:20, Steve D'Aprano wrote:
>> On Thu, 17 Nov 2016 10:37 pm, BartC wrote:
> 
>>> (I don't know how to disassemble code outside a function, not from
>>> inside the same program. Outside it might be: 'python -m dis file.py')
> 
>> In the most recent versions of Python, dis.dis() will also accept a string:
>> 
>> py> dis.dis('y = x + 1')
>>  1           0 LOAD_NAME                0 (x)
>>              3 LOAD_CONST               0 (1)
>>              6 BINARY_ADD
>>              7 STORE_NAME               1 (y)
>>             10 LOAD_CONST               1 (None)
>>             13 RETURN_VALUE
> 
> 
> Py2 gave me (for "y=x+1"):
> 
>          0 SETUP_EXCEPT    30781 (to 30784)
>          3 STORE_SLICE+3
>          4 <49>
> 
> Py3.4 works as you say but after that result I was disinclined to take it further!
> 
> -- 
> Bartc
> -- 
> https://mail.python.org/mailman/listinfo/python-list


From tjreedy at udel.edu  Thu Nov 17 12:08:58 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 17 Nov 2016 12:08:58 -0500
Subject: How to test for type or instance of dict_values?
In-Reply-To: <o0kgh9$pd6$1@blaine.gmane.org>
References: <o0k7j9$29r$1@blaine.gmane.org> <o0k8c1$g3o$1@blaine.gmane.org>
 <o0kgh9$pd6$1@blaine.gmane.org>
Message-ID: <o0ko78$3um$1@blaine.gmane.org>

On 11/17/2016 9:57 AM, Thorsten Kampe wrote:

> The code in question is part of an attempt to get the dimensions of
> multi-dimensional lists, the `isinstance` is there in order to
> exclude strings.

You can do the exclusion directly.

>
> """
> def dim(seq):
>     dimension = []
>     while isinstance(seq, (list, tuple)):

     while not isinstance(seq, str) # or (str, bytes, ...)

>         dimension.append(len(seq))
>         try:
>             seq = seq[0]
>         except IndexError:  # sequence is empty
>             break
>     return dimension
> """
>
> Thorsten
>


-- 
Terry Jan Reedy



From tjreedy at udel.edu  Thu Nov 17 12:41:54 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 17 Nov 2016 12:41:54 -0500
Subject: What exactly is a python variable?
In-Reply-To: <o0jfql$cn$1@dont-email.me>
References: <o0jfql$cn$1@dont-email.me>
Message-ID: <o0kq50$8cm$1@blaine.gmane.org>

On 11/17/2016 12:40 AM, Veek M wrote:
> In C:
> int x = 10;
> results in storage being allocated and type and location are fixed for
> the life of the program.
>
> In Python,
> x = 10
>
> causes an object '10' to be created but how exactly is 'x' handled?
> Symbol Table lookup at compile time?

Modules and class names are set into and gotten from a symbol table at 
*runtime* (except that module globals start with some initial reserved 
names).  Globals() returns the modules symbol table, which is a dict. 
To see the initialization, execute "print(globals())" on startup.  For 
CPython 3.6.0b3 executed via IDLE's shell:

 >>> globals()
{'__name__': '__main__',
  '__doc__': None,
  '__package__': None,
  '__loader__': <class '_frozen_importlib.BuiltinImporter'>,
  '__spec__': None,
  '__annotations__': {},
  '__builtins__': <module 'builtins' (built-in)>}

Class dicts are created at runtime and also have default entries.

 >>> class C: pass

 >>> print(C.__dict__)
{'__module__': '__main__', '__dict__': <attribute '__dict__' of 'C' 
objects>, '__weakref__': <attribute '__weakref__' of 'C' objects>, 
'__doc__': None}

 >>> print(dir(C))  # include attributes inherited from object.
['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', 
'__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', 
'__init__', '__init_subclass__', '__le__', '__lt__', '__module__', 
'__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', 
'__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__']

> Is every 'x' being substituted out of existence?

In CPython, function local names are replaced by int indexes into a list 
of objects, but still 'exist'.  They are kept with the code object as a 
list.  It is used for printing locals() and for disassembly output.  The 
latter is deceptive in this regard if you do not know that the names are 
substituted back for the indexes).

The local namespace, an attribute code does not have any default entries.

 >>> def f(): print(locals())

 >>> f()
{}

The code object itself and the function object that wraps a code object 
both have numerous special-name attributes.

-- 
Terry Jan Reedy



From rgaddi at highlandtechnology.invalid  Thu Nov 17 13:12:44 2016
From: rgaddi at highlandtechnology.invalid (Rob Gaddi)
Date: Thu, 17 Nov 2016 18:12:44 -0000 (UTC)
Subject: Access to the caller's globals, not your own
References: <582949b2$0$11109$c3e8da3@news.astraweb.com>
 <o0e4fp$tjb$1@dont-email.me> <582c1a73$0$11119$c3e8da3@news.astraweb.com>
 <o0i6do$786$1@dont-email.me>
 <582d3d81$0$1511$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o0krur$gbq$1@dont-email.me>

Steven D'Aprano wrote:

> On Thursday 17 November 2016 04:52, Rob Gaddi wrote:
>
>>> import library
>>> result = library.make_spam(arg)
>>>
>>>
>>> versus:
>>>
>>> import library
>>> make_spam = library.make_library()
>>> result = make_spam(arg)
>>>
>>> What a drag.
>>>
>>>
>> 
>> And there you have it; an entire extra line at import time.
>
> Multiplied by every function that has a configurable default. Multiplied by 
> every module that imports any of those functions, *whether or not* they change 
> the default. If there are four functions, the user has to do this:
>
> import library
> make_spam = library.make_spam_factory()
> make_eggs = library.make_eggs_factory()
> make_cheese = library.make_cheese_factory()
> make_toast = library.make_toast_factory()
>
>
> before they can even begin to do what they actually want to do, which is make 
> spam, eggs, cheese etc. And yes, there's likely to be four or more of these 
> functions.
>

No, no no no.  You misunderstood the approach that I think most of us
were on, which is

  import library
  Library = library.Library()
  Library.SPAMPARAM = 'jam'

  i_want_spam_now = Library.make_spam(99)

One extra line for the import, plus one for each default setting they
want to override (which you're stuck with either way).  I suppose you
could even pass all those overrides in to the Library __init__, but now
we're in the weeds.

And again, for the ones that don't want to mess with the default params,
you can have:

  import library.defaultinstance as Library


-- 
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order.  See above to fix.


From pkpearson at nowhere.invalid  Thu Nov 17 14:19:08 2016
From: pkpearson at nowhere.invalid (Peter Pearson)
Date: 17 Nov 2016 19:19:08 GMT
Subject: Overlapping co-ordiantes of rectangles fail to print in python
References: <1a9f480d-7e21-4559-aa38-a18899ca0b22@googlegroups.com>
Message-ID: <e96e5cFe8dhU1@mid.individual.net>

On Thu, 17 Nov 2016 06:27:49 -0800 (PST), arunsocs at gmail.com wrote:
>  I am working with following code in which I am trying to output co
>  ordinates of overlapping rectangles.. However the code fails to
>  output the co ordinates. I am customizing the following code

[about 100 lines of code removed]

> Where I need to change the code to output all overlapping rectangles
> and its co ordinates?

First, insert a print statement to determine whether the problem is
  (a) no overlapping rectangles are being found, or
  (b) the overlapping rectangles that are found are not being printed.

If (a), proceed to investigate whether
  (a1) there are, in fact, no overlapping rectangles, or
  (a2) your code is failing to recognize overlapping rectangles.

And so forth . . .

-- 
To email me, substitute nowhere->runbox, invalid->com.


From reddopp at gmail.com  Thu Nov 17 14:36:24 2016
From: reddopp at gmail.com (Pedro Franco de Carvalho)
Date: Thu, 17 Nov 2016 17:36:24 -0200
Subject: Can signal.alarm be safely cleared in python?
Message-ID: <CAOfc1cHiV+cTmwAZkCbGGZ0X4g-BoK8nLS7trPHHuLSaN6juag@mail.gmail.com>

Assume a python program sets a handler function for the
`signal.SIGALRM` signal, and then schedules an alarm in T seconds with
`signal.alarm(T)`.

Assume the program later cancels any scheduled alarm with `signal.alarm(0)`.

Can the program safely assume that after the call to `signal.alarm(0)`
completes no alarm handling function will be called?

The reason I ask is that the signal module documentation [1] says that:

"A Python signal handler does not get executed inside the low-level
(C) signal handler. Instead, the low-level signal handler sets a flag
which tells the virtual machine to execute the corresponding Python
signal handler at a later point(for example at the next bytecode
instruction)."

Since this "later point" is not specified, could it be that a the
alarm is triggered and calls the low-level C handler shortly before
the call to `signal.alarm(0)`, but the python signal handler is only
executed at some point after this call?

[1]: https://docs.python.org/3.6/library/signal.html#execution-of-python-signal-handlers)


From jelena.tavcar at gmail.com  Thu Nov 17 14:59:54 2016
From: jelena.tavcar at gmail.com (Jelena Tavcar)
Date: Thu, 17 Nov 2016 20:59:54 +0100
Subject: Python does not start
Message-ID: <CAKgG=gzTrFUV8ZGJo-_rjnc-7B1rRn5_wHKD+NNgkUAtK7+PaQ@mail.gmail.com>

How do I find stdlib files?
Regards


From jzayatz08730 at yahoo.com  Thu Nov 17 15:16:16 2016
From: jzayatz08730 at yahoo.com (John Zayatz)
Date: Thu, 17 Nov 2016 20:16:16 +0000 (UTC)
Subject: modify screen pop up
References: <2129454960.2005344.1479413776500.ref@mail.yahoo.com>
Message-ID: <2129454960.2005344.1479413776500@mail.yahoo.com>

when running pycharm the modify
setup window keep coming on the screen. I have uninstalled and
reinstalled python and pycharm multiple times. Do you have a solution?
Thank You"


From martin.pyka at gmail.com  Thu Nov 17 15:47:13 2016
From: martin.pyka at gmail.com (martin.pyka at gmail.com)
Date: Thu, 17 Nov 2016 12:47:13 -0800 (PST)
Subject: python package: financial_life
Message-ID: <2de273c2-ce4c-4fc7-bdec-cf576f9b37d6@googlegroups.com>

Hi everybody,

I recently uploaded a project I am working on for some months. It is called financial_life and the purpose of it is to simulate monetary flows between different accounts (like normal bank accounts, loans etc.). The simulations let you explore different financial strategies and figure out their long term impact on your asset. 

While you would need a lot of auxiliary tables in excel to achieve the same thing, in financial_life almost every line of code contributes only to the description of the problem. Thereby, with comparatively few amount of code lines, pretty complex simulations can be defined.

https://github.com/MartinPyka/financial_life

I hope, this package is helpful for some of you.

Best,
Martin


From bc at freeuk.com  Thu Nov 17 15:50:48 2016
From: bc at freeuk.com (BartC)
Date: Thu, 17 Nov 2016 20:50:48 +0000
Subject: Overlapping co-ordiantes of rectangles fail to print in python
In-Reply-To: <1a9f480d-7e21-4559-aa38-a18899ca0b22@googlegroups.com>
References: <1a9f480d-7e21-4559-aa38-a18899ca0b22@googlegroups.com>
Message-ID: <o0l569$jpd$1@dont-email.me>

On 17/11/2016 14:27, arunsocs at gmail.com wrote:
>  I am working with following code in which I am trying to output co ordinates of overlapping rectangles.. However the code fails to output the co ordinates. I am customizing the following code This is the input
>
> 1.6 1.2 7.9 3.1
> 1.2 1.6 3.4 7.2
> 2.6 11.6 6.8 14.0
> 9.6 1.2 11.4 7.5
> 9.6 1.7 14.1 2.8

(Is there a line count at the start of the file?)

> sign = -1
> area = sum( map( lambda x: x.area, rectangles) )
>
> for i in range(2,len(rectangles)+1):
>     for rects in combinations( rectangles, i ):
>         intersections = [rects[0]]
>         rects = rects[1:]
>         for rectangle in rects:
>             newintersections = []
>             for otherR in intersections:
>                 newintersections.extend( rectangle.intersect(otherR) )
>
>             intersections = newintersections
>             print intersections
>
>         #intersectingArea = sum( map( lambda x: x.area, intersections ) )
>         #rea = area + (sign * intersectingArea)
>
>     sign = sign*-1
>
> Where I need to change the code to output all overlapping rectangles and its co ordinates?

I tried your code and got a bunch of output, mostly [] lines like this:

[]
[]
[]
[]
[]
[[Point(x=1.6, y=1.6),Point(x=3.4, y=3.1)]]
[]
[]
[[Point(x=1.6, y=1.6),Point(x=3.4, y=3.1)]]
[]

What output did you expect for this input data?

(I plotted your data and out of five rectangles A,B,C,D,E (not in the 
same order as your data), then A and B overlap, as do C and D, while E 
is by itself.

Did you want the output to be, for example, A and B, with the 
coordinates of the common region, and the same for C and D?

What happens if 3 or more rectangles overlap?

What about if A overlaps B; B overlaps C; C overlaps D; and D overlaps 
A? Suppose then that all those are contained within E?

You need some specifications. I assume all rectangles are horizontally 
aligned (they have to be with just 4 coordinates), and the input data 
normalised so that the top-left corner appears first.)

-- 
Bartc


From joel.goldstick at gmail.com  Thu Nov 17 17:07:03 2016
From: joel.goldstick at gmail.com (Joel Goldstick)
Date: Thu, 17 Nov 2016 17:07:03 -0500
Subject: Python does not start
In-Reply-To: <CAKgG=gzTrFUV8ZGJo-_rjnc-7B1rRn5_wHKD+NNgkUAtK7+PaQ@mail.gmail.com>
References: <CAKgG=gzTrFUV8ZGJo-_rjnc-7B1rRn5_wHKD+NNgkUAtK7+PaQ@mail.gmail.com>
Message-ID: <CAPM-O+xRSt1PsRf_Vk5nkxkJPz4r_yS2uqBuU8dgBPP0fAcUEQ@mail.gmail.com>

On Thu, Nov 17, 2016 at 2:59 PM, Jelena Tavcar <jelena.tavcar at gmail.com> wrote:
> How do I find stdlib files?
> Regards
> --
> https://mail.python.org/mailman/listinfo/python-list

First of all, your subject line doesn't seem related to your body
text.  Second of all, stdlib is a C thing as I recall, not python.
Can you be a bit more forthcoming about your issue?

-- 
Joel Goldstick
http://joelgoldstick.com/blog
http://cc-baseballstats.info/stats/birthdays


From Ross.Boylan at ucsf.edu  Thu Nov 17 18:04:56 2016
From: Ross.Boylan at ucsf.edu (Boylan, Ross)
Date: Thu, 17 Nov 2016 23:04:56 +0000
Subject: how  to control formatting of a namedtuple in a list
In-Reply-To: <F1F13E14A610474196571953929C020924ED02FF@ex08.net.ucsf.edu>
References: <F1F13E14A610474196571953929C020924ED02FF@ex08.net.ucsf.edu>
Message-ID: <F1F13E14A610474196571953929C020924ED0357@ex08.net.ucsf.edu>

Even after defining custom __str__ and __format__ methods they don't affect the display of objects when they are in a list.  Is there a way to change that, other than explicitly converting each list element to a string?

The last line of output below shows that when I format the list I get standard formatting of my objects instead of my custom format.

Code
#! /usr/bin/python3
from collections import namedtuple

class Foo(namedtuple("Foo", "x")):
    __slots__ = ()

    def __str__(self):
          return "foolish({})".format(self.x)

    def __format__(self, spec):
          return self.__str__()

f=Foo(4)
print(f)
print(str(f))
print("{}".format(f))
print("{}".format([f]))  # a list with one f

Output
foolish(4)
foolish(4)
foolish(4)
[Foo(x=4)]

I'm running Python 3.4.

Thanks.
Ross


From rosuav at gmail.com  Thu Nov 17 18:24:28 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 18 Nov 2016 10:24:28 +1100
Subject: how to control formatting of a namedtuple in a list
In-Reply-To: <F1F13E14A610474196571953929C020924ED0357@ex08.net.ucsf.edu>
References: <F1F13E14A610474196571953929C020924ED02FF@ex08.net.ucsf.edu>
 <F1F13E14A610474196571953929C020924ED0357@ex08.net.ucsf.edu>
Message-ID: <CAPTjJmqjDYbwXog2WgTBp=cDaHhpjgV0PhSB7QG6jRbEjhC6ow@mail.gmail.com>

On Fri, Nov 18, 2016 at 10:04 AM, Boylan, Ross <Ross.Boylan at ucsf.edu> wrote:
> Even after defining custom __str__ and __format__ methods they don't affect the display of objects when they are in a list.  Is there a way to change that, other than explicitly converting each list element to a string?
>

Yep! Inside a list, it's the repr that gets shown. So you should be
able to do this:

class Foo(namedtuple("Foo", "x")):
    def __repr__(self):
          return "foolish({})".format(self.x)

This will also affect the other forms - if you don't define __str__,
it'll use __repr__. So this should be all you need.

ChrisA


From Ross.Boylan at ucsf.edu  Thu Nov 17 18:49:44 2016
From: Ross.Boylan at ucsf.edu (Boylan, Ross)
Date: Thu, 17 Nov 2016 23:49:44 +0000
Subject: how to control formatting of a namedtuple in a list
In-Reply-To: <CAPTjJmqjDYbwXog2WgTBp=cDaHhpjgV0PhSB7QG6jRbEjhC6ow@mail.gmail.com>
References: <F1F13E14A610474196571953929C020924ED02FF@ex08.net.ucsf.edu>
 <F1F13E14A610474196571953929C020924ED0357@ex08.net.ucsf.edu>,
 <CAPTjJmqjDYbwXog2WgTBp=cDaHhpjgV0PhSB7QG6jRbEjhC6ow@mail.gmail.com>
Message-ID: <7c18b6f8-e9fd-4411-9d75-6275acd6a57a@EXHT04.net.ucsf.edu>

Thank you; I can confirm that overriding __repr__ makes the list display as I wanted.

The decision to use repr inside the list seems very odd, given the context, namely formatting something for display or looking for a simple string representation.  It seems more natural to me to use str or, if in a format, the default formatting all the way down.  Is there a good reason it's repr?

Ross
________________________________________
From: Python-list [python-list-bounces+ross.boylan=ucsf.edu at python.org] on behalf of Chris Angelico [rosuav at gmail.com]
Sent: Thursday, November 17, 2016 3:24 PM
To: python-list at python.org
Subject: Re: how to control formatting of a namedtuple in a list

On Fri, Nov 18, 2016 at 10:04 AM, Boylan, Ross <Ross.Boylan at ucsf.edu> wrote:
> Even after defining custom __str__ and __format__ methods they don't affect the display of objects when they are in a list.  Is there a way to change that, other than explicitly converting each list element to a string?
>

Yep! Inside a list, it's the repr that gets shown. So you should be
able to do this:

class Foo(namedtuple("Foo", "x")):
    def __repr__(self):
          return "foolish({})".format(self.x)

This will also affect the other forms - if you don't define __str__,
it'll use __repr__. So this should be all you need.

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list


From ned at nedbatchelder.com  Thu Nov 17 19:08:41 2016
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Thu, 17 Nov 2016 16:08:41 -0800 (PST)
Subject: how to control formatting of a namedtuple in a list
In-Reply-To: <mailman.17.1479426592.20410.python-list@python.org>
References: <F1F13E14A610474196571953929C020924ED02FF@ex08.net.ucsf.edu>
 <F1F13E14A610474196571953929C020924ED0357@ex08.net.ucsf.edu>,>
 <CAPTjJmqjDYbwXog2WgTBp=cDaHhpjgV0PhSB7QG6jRbEjhC6ow@mail.gmail.com>
 <7c18b6f8-e9fd-4411-9d75-6275acd6a57a@EXHT04.net.ucsf.edu>
 <mailman.17.1479426592.20410.python-list@python.org>
Message-ID: <046d0656-150c-4068-a00b-98c4845ff001@googlegroups.com>

On Thursday, November 17, 2016 at 6:50:07 PM UTC-5, Boylan, Ross wrote:
> Thank you; I can confirm that overriding __repr__ makes the list display as I wanted.
> 
> The decision to use repr inside the list seems very odd, given the context, namely formatting something for display or looking for a simple string representation.  It seems more natural to me to use str or, if in a format, the default formatting all the way down.  Is there a good reason it's repr?


I think of it as str is for customers, repr is for developers. Or, repr is
for nerds, str is for civilians, etc.  If you print a list, then you will
get square brackets and commas separating the display.  That is, you are
already getting a nerdy output. So the contents of the list are also
displayed in the nerdy way, with repr.

If you want a nice display of the contents, you also have to control the
display of the list itself, so you can do whatever you need, and use str
on the contents yourself.

--Ned.


From python at mrabarnett.plus.com  Thu Nov 17 19:09:19 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 18 Nov 2016 00:09:19 +0000
Subject: how to control formatting of a namedtuple in a list
In-Reply-To: <7c18b6f8-e9fd-4411-9d75-6275acd6a57a@EXHT04.net.ucsf.edu>
References: <F1F13E14A610474196571953929C020924ED02FF@ex08.net.ucsf.edu>
 <F1F13E14A610474196571953929C020924ED0357@ex08.net.ucsf.edu>
 <CAPTjJmqjDYbwXog2WgTBp=cDaHhpjgV0PhSB7QG6jRbEjhC6ow@mail.gmail.com>
 <7c18b6f8-e9fd-4411-9d75-6275acd6a57a@EXHT04.net.ucsf.edu>
Message-ID: <35d62493-dbb7-8cc0-ece5-cd9fa62b29b7@mrabarnett.plus.com>

On 2016-11-17 23:49, Boylan, Ross wrote:
> Thank you; I can confirm that overriding __repr__ makes the list display as I wanted.
>
> The decision to use repr inside the list seems very odd, given the context, namely formatting something for display or looking for a simple string representation.  It seems more natural to me to use str or, if in a format, the default formatting all the way down.  Is there a good reason it's repr?
>
 > Ross

Given a string, say:

 >>> s = 'foo'

str shows:

 >>> print(str(s))

whereas repr shows:

 >>> print(repr(s))
'foo'

If it was in a list, would you want it to show:

[foo]

or:

['foo']

?

> ________________________________________
> From: Python-list [python-list-bounces+ross.boylan=ucsf.edu at python.org] on behalf of Chris Angelico [rosuav at gmail.com]
> Sent: Thursday, November 17, 2016 3:24 PM
> To: python-list at python.org
> Subject: Re: how to control formatting of a namedtuple in a list
>
> On Fri, Nov 18, 2016 at 10:04 AM, Boylan, Ross <Ross.Boylan at ucsf.edu> wrote:
>> Even after defining custom __str__ and __format__ methods they don't affect the display of objects when they are in a list.  Is there a way to change that, other than explicitly converting each list element to a string?
>>
>
> Yep! Inside a list, it's the repr that gets shown. So you should be
> able to do this:
>
> class Foo(namedtuple("Foo", "x")):
>     def __repr__(self):
>           return "foolish({})".format(self.x)
>
> This will also affect the other forms - if you don't define __str__,
> it'll use __repr__. So this should be all you need.
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>



From ethan at stoneleaf.us  Thu Nov 17 19:18:06 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 17 Nov 2016 16:18:06 -0800
Subject: how to control formatting of a namedtuple in a list
In-Reply-To: <35d62493-dbb7-8cc0-ece5-cd9fa62b29b7@mrabarnett.plus.com>
References: <F1F13E14A610474196571953929C020924ED02FF@ex08.net.ucsf.edu>
 <F1F13E14A610474196571953929C020924ED0357@ex08.net.ucsf.edu>
 <CAPTjJmqjDYbwXog2WgTBp=cDaHhpjgV0PhSB7QG6jRbEjhC6ow@mail.gmail.com>
 <7c18b6f8-e9fd-4411-9d75-6275acd6a57a@EXHT04.net.ucsf.edu>
 <35d62493-dbb7-8cc0-ece5-cd9fa62b29b7@mrabarnett.plus.com>
Message-ID: <582E48BE.500@stoneleaf.us>

On 11/17/2016 04:09 PM, MRAB wrote:
> On 2016-11-17 23:49, Boylan, Ross wrote:

>> Thank you; I can confirm that overriding __repr__ makes the list display as I wanted.
>>
>> The decision to use repr inside the list seems very odd, given the context, namely formatting something for display or looking for a simple string representation.  It seems more natural to me to use str or, if in a format, the default formatting all the way down.  Is there a good reason it's repr?
>
> Given a string, say:
>
> >>> s = 'foo'
>
> str shows:
>
> >>> print(str(s))
>
> whereas repr shows:
>
> >>> print(repr(s))
> 'foo'
>
> If it was in a list, would you want it to show:
>
> [foo]
>
> or:
>
> ['foo']
>
> ?

Another example:

  >>> foo = 'ham, eggs, cheese'
  >>> bar = 'bacon, toast'

if list used str instead of repr:

  >>> print(list(foo, bar))
  [ham, eegs, cheese, bacon, toast]

How many items are in that list?  (Hint: it isn't 5. ;)

--
~Ethan~


From tjreedy at udel.edu  Thu Nov 17 19:33:38 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 17 Nov 2016 19:33:38 -0500
Subject: Python does not start
In-Reply-To: <CAKgG=gzTrFUV8ZGJo-_rjnc-7B1rRn5_wHKD+NNgkUAtK7+PaQ@mail.gmail.com>
References: <CAKgG=gzTrFUV8ZGJo-_rjnc-7B1rRn5_wHKD+NNgkUAtK7+PaQ@mail.gmail.com>
Message-ID: <o0li90$lub$1@blaine.gmane.org>

On 11/17/2016 2:59 PM, Jelena Tavcar wrote:
> How do I find stdlib files?

Python coded stdlib files are, at least on Windows, in <install_dir>/Lib

-- 
Terry Jan Reedy



From steve+python at pearwood.info  Thu Nov 17 19:47:02 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Fri, 18 Nov 2016 11:47:02 +1100
Subject: What exactly is a python variable?
References: <o0jfql$cn$1@dont-email.me> <o0k4p2$r0e$1@dont-email.me>
 <582da094$0$1590$c3e8da3$5496439d@news.astraweb.com>
 <o0kao0$f04$1@dont-email.me>
Message-ID: <582e4f88$0$22141$c3e8da3$5496439d@news.astraweb.com>

On Fri, 18 Nov 2016 12:19 am, BartC wrote:

> On 17/11/2016 12:20, Steve D'Aprano wrote:
>> On Thu, 17 Nov 2016 10:37 pm, BartC wrote:
> 
>>> (I don't know how to disassemble code outside a function, not from
>>> inside the same program. Outside it might be: 'python -m dis file.py')
> 
>> In the most recent versions of Python, dis.dis() will also accept a
>> string:
>>
>> py> dis.dis('y = x + 1')
>>   1           0 LOAD_NAME                0 (x)
>>               3 LOAD_CONST               0 (1)
>>               6 BINARY_ADD
>>               7 STORE_NAME               1 (y)
>>              10 LOAD_CONST               1 (None)
>>              13 RETURN_VALUE
> 
> 
> Py2 gave me (for "y=x+1"):
> 
>            0 SETUP_EXCEPT    30781 (to 30784)
>            3 STORE_SLICE+3
>            4 <49>
> 
> Py3.4 works as you say but after that result I was disinclined to take
> it further!


You may have missed the bit where I said "In the most recent versions".
Python 2.7 will be interpreting the string "y=x+1" as compiled byte-code,
and disassembling it into junk.

If in doubt, just use the compile() function first.



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From rosuav at gmail.com  Thu Nov 17 19:58:08 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 18 Nov 2016 11:58:08 +1100
Subject: What exactly is a python variable?
In-Reply-To: <582e4f88$0$22141$c3e8da3$5496439d@news.astraweb.com>
References: <o0jfql$cn$1@dont-email.me> <o0k4p2$r0e$1@dont-email.me>
 <582da094$0$1590$c3e8da3$5496439d@news.astraweb.com>
 <o0kao0$f04$1@dont-email.me>
 <582e4f88$0$22141$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPTjJmpeTv8EE9mJ0DH6=aHCnWqNDqNoWucULS+HSnEOi5Kf2g@mail.gmail.com>

On Fri, Nov 18, 2016 at 11:47 AM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> You may have missed the bit where I said "In the most recent versions".
> Python 2.7 will be interpreting the string "y=x+1" as compiled byte-code,
> and disassembling it into junk.
>

Heh. That's a side benefit of the Py3 bytes/unicode split - you can
instantly distinguish between program text and compiled bytes. :)

ChrisA


From bc at freeuk.com  Thu Nov 17 20:34:55 2016
From: bc at freeuk.com (BartC)
Date: Fri, 18 Nov 2016 01:34:55 +0000
Subject: What exactly is a python variable?
In-Reply-To: <582e4f88$0$22141$c3e8da3$5496439d@news.astraweb.com>
References: <o0jfql$cn$1@dont-email.me> <o0k4p2$r0e$1@dont-email.me>
 <582da094$0$1590$c3e8da3$5496439d@news.astraweb.com>
 <o0kao0$f04$1@dont-email.me>
 <582e4f88$0$22141$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o0llqv$bfo$1@dont-email.me>

On 18/11/2016 00:47, Steve D'Aprano wrote:
> On Fri, 18 Nov 2016 12:19 am, BartC wrote:
>
>> On 17/11/2016 12:20, Steve D'Aprano wrote:

>>> In the most recent versions of Python, dis.dis() will also accept a
>>> string:
>>>
>>> py> dis.dis('y = x + 1')
>>>   1           0 LOAD_NAME                0 (x)
>>>               3 LOAD_CONST               0 (1)
>>>               6 BINARY_ADD
>>>               7 STORE_NAME               1 (y)
>>>              10 LOAD_CONST               1 (None)
>>>              13 RETURN_VALUE
>>
>>
>> Py2 gave me (for "y=x+1"):
>>
>>            0 SETUP_EXCEPT    30781 (to 30784)
>>            3 STORE_SLICE+3
>>            4 <49>
>>
>> Py3.4 works as you say but after that result I was disinclined to take
>> it further!
>
>
> You may have missed the bit where I said "In the most recent versions".
> Python 2.7 will be interpreting the string "y=x+1" as compiled byte-code,
> and disassembling it into junk.

I did my tests ('dis.dis("x=10")') before your post and before my 
original post. Py2 generated nonsense so I tried something else.

That Py2's dis.dis() accepts a string argument but treats it as compiled 
byte-code sounds like a bug. Unless it's a feature.

-- 
Bartc



From Ross.Boylan at ucsf.edu  Thu Nov 17 20:39:07 2016
From: Ross.Boylan at ucsf.edu (Boylan, Ross)
Date: Fri, 18 Nov 2016 01:39:07 +0000
Subject: how to control formatting of a namedtuple in a list
In-Reply-To: <582E48BE.500@stoneleaf.us>
References: <F1F13E14A610474196571953929C020924ED02FF@ex08.net.ucsf.edu>
 <F1F13E14A610474196571953929C020924ED0357@ex08.net.ucsf.edu>
 <CAPTjJmqjDYbwXog2WgTBp=cDaHhpjgV0PhSB7QG6jRbEjhC6ow@mail.gmail.com>
 <7c18b6f8-e9fd-4411-9d75-6275acd6a57a@EXHT04.net.ucsf.edu>
 <35d62493-dbb7-8cc0-ece5-cd9fa62b29b7@mrabarnett.plus.com>,
 <582E48BE.500@stoneleaf.us>
Message-ID: <5f142e8a-62cc-43a2-9634-e3c6032cddc3@EXHT01.net.ucsf.edu>

Actually,
>> print(list(foo, bar))
Traceback (most recent call last):                                                                                              
  File "<stdin>", line 1, in <module>                                                                                           
TypeError: list() takes at most 1 argument (2 given)       

Though                                                                     
>>> [foo, bar]
['ham, eggs, cheese', 'bacon, toast'] 
which admittedly would be confusing if the strings weren't quoted. But display, or formatted values, are not intended to allow reconstruction of the objects; that's what repr is for. 

The argument that the display of list is already low-level, and so everything inside the list should be displayed low-level, seems a bit of a stretch.

Overriding repr is serviceable for me, but it requires violating the intended semantics of repr, namely that the result could be converted back to the original object. I'm trying to get a display that has only some of the information in the object.  My understanding is that str is supposed to provide that.

At any rate, I agree there are reasons to use repr inside a list.

Ross
________________________________________
From: Python-list [python-list-bounces+ross.boylan=ucsf.edu at python.org] on behalf of Ethan Furman [ethan at stoneleaf.us]
Sent: Thursday, November 17, 2016 4:18 PM
To: python-list at python.org
Subject: Re: how to control formatting of a namedtuple in a list

On 11/17/2016 04:09 PM, MRAB wrote:
> On 2016-11-17 23:49, Boylan, Ross wrote:

>> Thank you; I can confirm that overriding __repr__ makes the list display as I wanted.
>>
>> The decision to use repr inside the list seems very odd, given the context, namely formatting something for display or looking for a simple string representation.  It seems more natural to me to use str or, if in a format, the default formatting all the way down.  Is there a good reason it's repr?
>
> Given a string, say:
>
> >>> s = 'foo'
>
> str shows:
>
> >>> print(str(s))
>
> whereas repr shows:
>
> >>> print(repr(s))
> 'foo'
>
> If it was in a list, would you want it to show:
>
> [foo]
>
> or:
>
> ['foo']
>
> ?

Another example:

  >>> foo = 'ham, eggs, cheese'
  >>> bar = 'bacon, toast'

if list used str instead of repr:

  >>> print(list(foo, bar))
  [ham, eegs, cheese, bacon, toast]

How many items are in that list?  (Hint: it isn't 5. ;)

--
~Ethan~
--
https://mail.python.org/mailman/listinfo/python-list


From ned at nedbatchelder.com  Thu Nov 17 20:52:10 2016
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Thu, 17 Nov 2016 17:52:10 -0800 (PST)
Subject: What exactly is a python variable?
In-Reply-To: <o0llqv$bfo$1@dont-email.me>
References: <o0jfql$cn$1@dont-email.me> <o0k4p2$r0e$1@dont-email.me>
 <582da094$0$1590$c3e8da3$5496439d@news.astraweb.com>
 <o0kao0$f04$1@dont-email.me>
 <582e4f88$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <o0llqv$bfo$1@dont-email.me>
Message-ID: <45752e07-e26e-4267-9704-e609b4e238cc@googlegroups.com>

On Thursday, November 17, 2016 at 8:35:15 PM UTC-5, BartC wrote:
> That Py2's dis.dis() accepts a string argument but treats it as compiled 
> byte-code sounds like a bug. Unless it's a feature.

In Python 2, plain-old strings are byte-strings, so there's no way for
dis.dis to distinguish between a string literal full of code, and a
byte-string full of bytecode.

So it's not a bug, and it also is not a feature. It just is.

--Ned.


From jfong at ms4.hinet.net  Thu Nov 17 20:56:29 2016
From: jfong at ms4.hinet.net (jfong at ms4.hinet.net)
Date: Thu, 17 Nov 2016 17:56:29 -0800 (PST)
Subject: help on "from deen import *" vs. "import deen"
In-Reply-To: <582d6467$0$1505$c3e8da3$5496439d@news.astraweb.com>
References: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>
 <ca14fafb-2226-493e-ada5-e850a3c74f48@googlegroups.com>
 <e0478664-9610-4d83-8f58-583682d11fcd@googlegroups.com>
 <bb816fbf-1df7-522d-bab7-9f7a65deaef3@mrabarnett.plus.com>
 <mailman.147.1479180690.30068.python-list@python.org>
 <1a6f991d-53c1-4c79-b81b-e804b01be653@googlegroups.com>
 <144b0566-6738-bd37-cd1c-1585e70761d2@gmail.com>
 <9b6ad48d-ee65-bca6-6fa7-4176ddaae83e@lucidity.plus.com>
 <mailman.153.1479248175.30068.python-list@python.org>
 <582ba948$0$1589$c3e8da3$5496439d@news.astraweb.com>
 <5dcc9d11-a336-41f5-bed6-5e073f614bd0@googlegroups.com>
 <582d2cae$0$1517$c3e8da3$5496439d@news.astraweb.com>
 <f1ae52cf-6582-43ac-b65b-c7bad2b35f15@googlegroups.com>
 <582d6467$0$1505$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <e6426da0-ea3f-43f5-9b95-af87236458e9@googlegroups.com>

Steven D'Aprano at 2016/11/17 4:04:04PM wrote:
> The most important thing you should learn from this thread is:
> 
> - avoid using "from module import *" as it is usually more trouble 
>   than it is worth.
> 
> 
> It is confusing and leads to more problems than it solves. If Python was being 
> invented now, rather than 20 years ago, I would expect that there would be no 
> "import *" in the language.

It's the greatest advice I ever had in Python:-)

--Jach



From nathan.ernst at gmail.com  Thu Nov 17 21:14:46 2016
From: nathan.ernst at gmail.com (Nathan Ernst)
Date: Thu, 17 Nov 2016 20:14:46 -0600
Subject: help on "from deen import *" vs. "import deen"
In-Reply-To: <e6426da0-ea3f-43f5-9b95-af87236458e9@googlegroups.com>
References: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>
 <ca14fafb-2226-493e-ada5-e850a3c74f48@googlegroups.com>
 <e0478664-9610-4d83-8f58-583682d11fcd@googlegroups.com>
 <bb816fbf-1df7-522d-bab7-9f7a65deaef3@mrabarnett.plus.com>
 <mailman.147.1479180690.30068.python-list@python.org>
 <1a6f991d-53c1-4c79-b81b-e804b01be653@googlegroups.com>
 <144b0566-6738-bd37-cd1c-1585e70761d2@gmail.com>
 <9b6ad48d-ee65-bca6-6fa7-4176ddaae83e@lucidity.plus.com>
 <mailman.153.1479248175.30068.python-list@python.org>
 <582ba948$0$1589$c3e8da3$5496439d@news.astraweb.com>
 <5dcc9d11-a336-41f5-bed6-5e073f614bd0@googlegroups.com>
 <582d2cae$0$1517$c3e8da3$5496439d@news.astraweb.com>
 <f1ae52cf-6582-43ac-b65b-c7bad2b35f15@googlegroups.com>
 <582d6467$0$1505$c3e8da3$5496439d@news.astraweb.com>
 <e6426da0-ea3f-43f5-9b95-af87236458e9@googlegroups.com>
Message-ID: <CAPjBxg9ygZpXxoeCCju9ancXZ5ZQ+k5GpbN0Xrc6SX_Ve9ZyZA@mail.gmail.com>

I would also toss in there: never name a script test.py. Causes nothing but
trouble, at least in python2.

On Nov 17, 2016 8:01 PM, <jfong at ms4.hinet.net> wrote:

> Steven D'Aprano at 2016/11/17 4:04:04PM wrote:
> > The most important thing you should learn from this thread is:
> >
> > - avoid using "from module import *" as it is usually more trouble
> >   than it is worth.
> >
> >
> > It is confusing and leads to more problems than it solves. If Python was
> being
> > invented now, rather than 20 years ago, I would expect that there would
> be no
> > "import *" in the language.
>
> It's the greatest advice I ever had in Python:-)
>
> --Jach
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From jfong at ms4.hinet.net  Thu Nov 17 21:23:39 2016
From: jfong at ms4.hinet.net (jfong at ms4.hinet.net)
Date: Thu, 17 Nov 2016 18:23:39 -0800 (PST)
Subject: help on "from deen import *" vs. "import deen"
In-Reply-To: <mailman.7.1479397099.20410.python-list@python.org>
References: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>
 <ca14fafb-2226-493e-ada5-e850a3c74f48@googlegroups.com>
 <e0478664-9610-4d83-8f58-583682d11fcd@googlegroups.com>
 <bb816fbf-1df7-522d-bab7-9f7a65deaef3@mrabarnett.plus.com>
 <mailman.147.1479180690.30068.python-list@python.org>
 <1a6f991d-53c1-4c79-b81b-e804b01be653@googlegroups.com>
 <144b0566-6738-bd37-cd1c-1585e70761d2@gmail.com>
 <mailman.151.1479221023.30068.python-list@python.org>
 <80870bc5-5975-43e0-bf23-dccb1f628d6f@googlegroups.com>
 <5d48ae5f-17f1-fca7-1239-d22c2950363c@gmail.com>
 <mailman.154.1479266093.30068.python-list@python.org>
 <c4c12dd3-7d2d-4a87-bdbc-0e752dbf5318@googlegroups.com>
 <ababec33-ad77-60f4-367f-aa89aad22230@gmail.com>
 <mailman.7.1479397099.20410.python-list@python.org>
Message-ID: <023fdec4-3630-4030-a58b-24028dde4b41@googlegroups.com>

Michael Torrie at 2016/11/17 11:38:32PM wrote:
> Like I said, whether the names you use are appropriate is completely up
> to you.  But this statement seems to imply you're still not getting it
> and still thinking of variables as boxes like they are in other
> languages, rather than labels that can only point to one thing at a
> time.

As a knowledge, I do know the re-bind tricky Python had on its variable, and that's mostly I stumbled so far when writing Python codes.

> Python's variables are different from other languages, but in an
> understandable way.  

Unfortunately it's also different from human language.

> As Dennis said, understanding Python variables,
> though not difficult, needs to be second nature to you or you'll be
> fighting Python and hating the experience.

That's the point: to build a second nature.

But it won't be easy because of we use human language used everyday:-(

--Jach





From torriem at gmail.com  Thu Nov 17 22:02:52 2016
From: torriem at gmail.com (Michael Torrie)
Date: Thu, 17 Nov 2016 20:02:52 -0700
Subject: help on "from deen import *" vs. "import deen"
In-Reply-To: <023fdec4-3630-4030-a58b-24028dde4b41@googlegroups.com>
References: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>
 <ca14fafb-2226-493e-ada5-e850a3c74f48@googlegroups.com>
 <e0478664-9610-4d83-8f58-583682d11fcd@googlegroups.com>
 <bb816fbf-1df7-522d-bab7-9f7a65deaef3@mrabarnett.plus.com>
 <mailman.147.1479180690.30068.python-list@python.org>
 <1a6f991d-53c1-4c79-b81b-e804b01be653@googlegroups.com>
 <144b0566-6738-bd37-cd1c-1585e70761d2@gmail.com>
 <mailman.151.1479221023.30068.python-list@python.org>
 <80870bc5-5975-43e0-bf23-dccb1f628d6f@googlegroups.com>
 <5d48ae5f-17f1-fca7-1239-d22c2950363c@gmail.com>
 <mailman.154.1479266093.30068.python-list@python.org>
 <c4c12dd3-7d2d-4a87-bdbc-0e752dbf5318@googlegroups.com>
 <ababec33-ad77-60f4-367f-aa89aad22230@gmail.com>
 <mailman.7.1479397099.20410.python-list@python.org>
 <023fdec4-3630-4030-a58b-24028dde4b41@googlegroups.com>
Message-ID: <5ee9f8ce-e3a6-8163-02da-f5d928d219e1@gmail.com>

On 11/17/2016 07:23 PM, jfong at ms4.hinet.net wrote:
>> Python's variables are different from other languages, but in an
>> understandable way.  
> 
> Unfortunately it's also different from human language.

How so? I don't find this to be true at all.


From valkrem at yahoo.com  Thu Nov 17 22:05:14 2016
From: valkrem at yahoo.com (Val Krem)
Date: Fri, 18 Nov 2016 03:05:14 +0000 (UTC)
Subject: key and ..
References: <787076522.2584981.1479438314739.ref@mail.yahoo.com>
Message-ID: <787076522.2584981.1479438314739@mail.yahoo.com>



Hi all,
Sorry for asking such a basic question butI am trying  to merge two files(file1 and file2) and do some stuff. Merge the two files by the first column(key). Here is the description of files and what I would like to do.


file1

key c1   c2
1  759   939
2  345 154571
3  251 350711
4 3749  22159
5  676  76953
6   46    756


file2
key  p1    p2
1   759    939
2   345 154571
3   251 350711
4  3915  23254
5  7676  77953
7   256   4562

create file3
a) merge the two files by (key) that exit in  file1 and file2
b) create two variables dcp1 = c1- p1 and dcp2= c2-p2
c) sort file3 by dcp2(descending) and output

create file4:-  which exist in file1 but not in file2
create file5:-  that exist in file2 but not in file1;


Desired output files

file3
key   c1    c2     p1  p2     dcp1   dcp2
4   3749  22159  3915  23254  -166  -1095
5    676  76953  7676  77953 -7000  -1000
1    759    939   759    939     0      0
2    345 154571   345 154571     0      0
3    251 350711   251 350711     0      0

file4
key c1   p1
6   46   756

file5
key p1   p2
7  256  4562



Thank you in advance


From jfong at ms4.hinet.net  Thu Nov 17 22:41:28 2016
From: jfong at ms4.hinet.net (jfong at ms4.hinet.net)
Date: Thu, 17 Nov 2016 19:41:28 -0800 (PST)
Subject: help on "from deen import *" vs. "import deen"
In-Reply-To: <mailman.24.1479438177.20410.python-list@python.org>
References: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>
 <ca14fafb-2226-493e-ada5-e850a3c74f48@googlegroups.com>
 <e0478664-9610-4d83-8f58-583682d11fcd@googlegroups.com>
 <bb816fbf-1df7-522d-bab7-9f7a65deaef3@mrabarnett.plus.com>
 <mailman.147.1479180690.30068.python-list@python.org>
 <1a6f991d-53c1-4c79-b81b-e804b01be653@googlegroups.com>
 <144b0566-6738-bd37-cd1c-1585e70761d2@gmail.com>
 <mailman.151.1479221023.30068.python-list@python.org>
 <80870bc5-5975-43e0-bf23-dccb1f628d6f@googlegroups.com>
 <5d48ae5f-17f1-fca7-1239-d22c2950363c@gmail.com>
 <mailman.154.1479266093.30068.python-list@python.org>
 <c4c12dd3-7d2d-4a87-bdbc-0e752dbf5318@googlegroups.com>
 <ababec33-ad77-60f4-367f-aa89aad22230@gmail.com>
 <mailman.7.1479397099.20410.python-list@python.org>
 <023fdec4-3630-4030-a58b-24028dde4b41@googlegroups.com>
 <5ee9f8ce-e3a6-8163-02da-f5d928d219e1@gmail.com>
 <mailman.24.1479438177.20410.python-list@python.org>
Message-ID: <90551ab1-c0b9-48db-85c3-c9487926b3f7@googlegroups.com>

Michael Torrie at 2016/11/18 11:03:12AM wrote:
> >> Python's variables are different from other languages, but in an
> >> understandable way.  
> > 
> > Unfortunately it's also different from human language.
> 
> How so? I don't find this to be true at all.

The fact that most novices will stumble on Python variable many times until it becomes his "second nature" proves it's different from the human language:-)

--Jach


From walters.justin01 at gmail.com  Thu Nov 17 23:06:46 2016
From: walters.justin01 at gmail.com (justin walters)
Date: Thu, 17 Nov 2016 20:06:46 -0800
Subject: key and ..
In-Reply-To: <787076522.2584981.1479438314739@mail.yahoo.com>
References: <787076522.2584981.1479438314739.ref@mail.yahoo.com>
 <787076522.2584981.1479438314739@mail.yahoo.com>
Message-ID: <CAO1D73HYu-CY68xpRa26w_apW+DHd0NYErjcj98ozZ+u-JyUpQ@mail.gmail.com>

On Thu, Nov 17, 2016 at 7:05 PM, Val Krem via Python-list <
python-list at python.org> wrote:

>
>
> Hi all,
> Sorry for asking such a basic question butI am trying  to merge two
> files(file1 and file2) and do some stuff. Merge the two files by the first
> column(key). Here is the description of files and what I would like to do.
>
>
> file1
>
> key c1   c2
> 1  759   939
> 2 345 154571
> 3  251 350711
> 4 3749  22159
> 5  676  76953
> 6   46    756
>
>
> file2
> key  p1    p2
> 1   759    939
> 2   345 154571
> 3   251 350711
> 4  3915  23254
> 5  7676  77953
> 7   256   4562
>
> create file3
> a) merge the two files by (key) that exit in  file1 and file2
> b) create two variables dcp1 = c1- p1 and dcp2= c2-p2
> c) sort file3 by dcp2(descending) and output
>
> create file4:-  which exist in file1 but not in file2
> create file5:-  that exist in file2 but not in file1;
>
>
> Desired output files
>
> file3
> key   c1    c2     p1  p2     dcp1   dcp2
> 4   3749  22159  3915  23254  -166  -1095
> 5    676  76953  7676  77953 -7000  -1000
> 1    759    939   759    939     0      0
> 2    345 154571   345 154571     0      0
> 3    251 350711   251 350711     0      0
>
> file4
> key c1   p1
> 6   46   756
>
> file5
> key p1   p2
> 7  256  4562
>
>
>
> Thank you in advance
> --
> https://mail.python.org/mailman/listinfo/python-list
>

1. Take each file and read it using file.open() declaring a variable to
store the string.
2. Use list.split('\n') to split the file into an array of lines.
3. Build a list of dictionaries by splitting each line at whitespace and
calling int() on the values
    of each column for each file.
4. Do what you have to do math wise between each dict storing the values in
a new dict. You can
    write this out directly to the file or append it to a new list.
5. Use file.open() to write the resulting lines to a new file.
6. transform one of the lists into a set and use set.difference() or
set.intersection() to create
    a new list. This list will be unordered by default, so you may want to
run it through
    sorted(set, key=lambda row: row['key']).
7. repeat step 5 above to write out to file 4 and 5. no need to transform
the list into a set again.
    Just find the difference/interference again.

This isn't the fastest or most efficient way of doing it, but it is
probably the most straight forward.
If these files are quite large you may want to take a different approach in
the interest of performance
and memory. If you don't want to use dicts, you should have no problem
substituting tuples or
nested lists.

The whole thing could be made into a generator as well.

Basically, there are a lot of ways to approach this.

Hope that helped at least a little bit.


From torriem at gmail.com  Thu Nov 17 23:14:07 2016
From: torriem at gmail.com (Michael Torrie)
Date: Thu, 17 Nov 2016 21:14:07 -0700
Subject: help on "from deen import *" vs. "import deen"
In-Reply-To: <90551ab1-c0b9-48db-85c3-c9487926b3f7@googlegroups.com>
References: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>
 <ca14fafb-2226-493e-ada5-e850a3c74f48@googlegroups.com>
 <e0478664-9610-4d83-8f58-583682d11fcd@googlegroups.com>
 <bb816fbf-1df7-522d-bab7-9f7a65deaef3@mrabarnett.plus.com>
 <mailman.147.1479180690.30068.python-list@python.org>
 <1a6f991d-53c1-4c79-b81b-e804b01be653@googlegroups.com>
 <144b0566-6738-bd37-cd1c-1585e70761d2@gmail.com>
 <mailman.151.1479221023.30068.python-list@python.org>
 <80870bc5-5975-43e0-bf23-dccb1f628d6f@googlegroups.com>
 <5d48ae5f-17f1-fca7-1239-d22c2950363c@gmail.com>
 <mailman.154.1479266093.30068.python-list@python.org>
 <c4c12dd3-7d2d-4a87-bdbc-0e752dbf5318@googlegroups.com>
 <ababec33-ad77-60f4-367f-aa89aad22230@gmail.com>
 <mailman.7.1479397099.20410.python-list@python.org>
 <023fdec4-3630-4030-a58b-24028dde4b41@googlegroups.com>
 <5ee9f8ce-e3a6-8163-02da-f5d928d219e1@gmail.com>
 <mailman.24.1479438177.20410.python-list@python.org>
 <90551ab1-c0b9-48db-85c3-c9487926b3f7@googlegroups.com>
Message-ID: <e558b448-702c-c8c0-296a-ad60d6cca18d@gmail.com>

On 11/17/2016 08:41 PM, jfong at ms4.hinet.net wrote:
> The fact that most novices will stumble on Python variable many times
> until it becomes his "second nature" proves it's different from the
> human language:-)

The fact is that most novices don't stumble when dealing with Python
variables. The nature of name binding vs memory boxes does trip up
people occasionally, but most grasp it quickly and move on.

Since you are unwilling to prove your assertion or even provide a single
example (you did once speak of couches and chairs but your reference
made little sense), we can make no further progress.



From greg.ewing at canterbury.ac.nz  Thu Nov 17 23:40:48 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Fri, 18 Nov 2016 17:40:48 +1300
Subject: Error in webscraping problem
In-Reply-To: <1d10586d-e517-462c-8397-b18c56fbd48c@googlegroups.com>
References: <1d10586d-e517-462c-8397-b18c56fbd48c@googlegroups.com>
Message-ID: <e97f2iFlt2oU1@mid.individual.net>

stanleydasilva93 at gmail.com wrote:
> I am trying to solve the following problem.  Two numbers appear on a website.
> The user has to enter the gcd (greatest common divisor) and hit the submit
> button.  The catch is that the time limit is far too slow for any human
> processes -- it must be fully automated.

That's an extremely weird set of requirements. What is it,
some kind of anti-captcha, proving that you're *not* a
human?

> print firstNumber # Looks sensible -- first number probably correct
> print secondNumber # Looks sensible -- also checked
> print solution # Is indeed the correct gcd
> res = br.submit()
> 
> content = res.read()
> with open("FinalResults.html", "w") as f:
>     f.write(content)
> # Unfortunately, I examine this file to find "Wrong Answer"

If the numbers are correct, there must be something wrong
with the way you're submitting the form.

Maybe the site is expecting some cookies or header values
that you're not sending?

Are you able to submit a result successfully by hand
using a browser? If so, you might be able to use Firefox's
debugging facilities to capture the exact response being
sent, including headers and cookies.

-- 
Greg


From ayush.agg90 at gmail.com  Fri Nov 18 00:29:03 2016
From: ayush.agg90 at gmail.com (Ayush Aggarwal)
Date: Fri, 18 Nov 2016 10:59:03 +0530
Subject: Unable to sniff outgoing traffic using raw sockets in python2.7
Message-ID: <CAKa7J-YUpVmO29Hi6kov0DB-w=DgF-4QGJKGgNYqr10XpHoMWw@mail.gmail.com>

Hello,

Following is my code :

#!/usr/bin/python

import socket
import struct
import binascii

rawSocket = socket.socket(socket.PF_PACKET,socket.SOCK_RAW,socket.htons(0x0800))
# use 0x0800 for IPv4 packets , 0x0003 is for sniffing all kinds of packets

while True:
        pkt= rawSocket.recvfrom(2048)

        ethernetHeader = pkt[0][0:14]

        pr = unicode(ethernetHeader, errors='replace')
        print pr
        eth_hdr = struct.unpack("!6s6s2s",ethernetHeader)

        print "Source MAC Address :" , binascii.hexlify(eth_hdr[1])
        print "Destination MAC Address : " , binascii.hexlify(eth_hdr[0])
        print "Protocol : " , binascii.hexlify(eth_hdr[2])
        ipHeader = pkt[0][14:34]
        ip_hdr = struct.unpack("!12s4s4s",ipHeader)
        print "Source ip ADDRESS : " + socket.inet_ntoa(ip_hdr[1])
        print "Destination IP Address: " + socket.inet_ntoa(ip_hdr[2])
        # initial part of the tcp header
        tcpHeader = pkt[0][34:54]
        tcp_hdr = struct.unpack("!HH16s",tcpHeader)
        print "Source Port ADDRESS : " ,tcp_hdr[0]
        print "Destination Port ADDRESS : " , tcp_hdr[1]


Issues :
1. Unable to capture any outgoing IPv4 traffic. I ran the sniff()
method in Scapy and it does capture the outgoing packets.

2. I am NOT USING PROMISCUOUS MODE , still most of the packes I am
receiving neither have my IP or MAC in either of the source or
destination fields.

3. Captured data is different from the one observed using Scapy or Wireshark.

Request you to kindly clarify these observations.

Thanks and Regards,
Ayush


From jobmattcon at gmail.com  Fri Nov 18 00:55:04 2016
From: jobmattcon at gmail.com (meInvent bbird)
Date: Thu, 17 Nov 2016 21:55:04 -0800 (PST)
Subject: how to simulate the situation in DNA evolution for finding the
 minimum population needed and minimum samples selected to mating in order to
 no extinction in any one of original species and new species
Message-ID: <156581c9-1684-462c-a7d9-0692c6083278@googlegroups.com>

how to simulate the situation in DNA evolution for finding the minimum population needed and minimum samples selected to mating in order to no extinction in any one of original species and new species

assume mating are randomly selected to become a couple,
how to keep species good which means no extinction in original species

i use i+j to get new species, 
but i do not know whether there is limit in evolution, i assume
limit is 7, then extra evolution will go back to past species and 
cycle again

import matplotlib.pyplot as plt
import random
dict = {}
dist = {}
maxnum = 5
allowedmax = 7
for i in range(1,allowedmax+1):
    dist[str(i)] = 0

rr = range (1,maxnum)
for i in range (1,maxnum):
    for j in range (1,maxnum):
        if i < j:
            print("(" +str(i) + "," + str(j) + ")");
            dict[str(i) + str(j)] = 1;
            dist[str(i+j)] = dist[str(i+j)] + 1
            if i+j > max(rr or [0]):
                 rr = rr + [i+j];

original = rr;
for numberofevolutions in range(1,10):
    rr2 = []
    samples = random.sample(original, len(original)-2)
    print("total rr")
    print(str(rr))
    print("samples")
    print(str(samples))
    for i in samples:
        for j in samples:
            if i < j:
                print("(" +str(i) + "," + str(j) + ")");
                if i+j > allowedmax:
                    dict[str(i) + str(j)] = (i+j) % allowedmax;
                    dist[str((i+j) % allowedmax)] = dist[str((i+j) % allowedmax)] + 1
                    if ((i+j) % allowedmax) > max(rr2 or [0]):
                        rr2 = rr2 + [((i+j) % allowedmax)];
                else:
                    dict[str(i) + str(j)] = i+j;
                    dist[str(i+j)] = dist[str(i+j)] + 1
                    if i+j > max(rr2 or [0]):
                        rr2 = rr2 + [i+j];
    temp = rr
    rr = rr2
    rr2 = temp

plt.bar(range(len(dist)), dist.values(), align='center')
plt.xticks(range(len(dist)), dist.keys())
plt.show()


From ben+python at benfinney.id.au  Fri Nov 18 01:51:03 2016
From: ben+python at benfinney.id.au (Ben Finney)
Date: Fri, 18 Nov 2016 17:51:03 +1100
Subject: What exactly is a python variable?
References: <o0jfql$cn$1@dont-email.me> <o0k4p2$r0e$1@dont-email.me>
 <582da094$0$1590$c3e8da3$5496439d@news.astraweb.com>
 <o0kao0$f04$1@dont-email.me>
 <582e4f88$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <o0llqv$bfo$1@dont-email.me>
 <45752e07-e26e-4267-9704-e609b4e238cc@googlegroups.com>
Message-ID: <85zikx46y0.fsf@benfinney.id.au>

Ned Batchelder <ned at nedbatchelder.com> writes:

> So it's not a bug, and it also is not a feature. It just is.

+1 QotW

-- 
 \        ?A free press is one where it's okay to state the conclusion |
  `\                      you're led to by the evidence.? ?Bill Moyers |
_o__)                                                                  |
Ben Finney



From dieter at handshake.de  Fri Nov 18 03:51:20 2016
From: dieter at handshake.de (dieter)
Date: Fri, 18 Nov 2016 09:51:20 +0100
Subject: Can signal.alarm be safely cleared in python?
References: <CAOfc1cHiV+cTmwAZkCbGGZ0X4g-BoK8nLS7trPHHuLSaN6juag@mail.gmail.com>
Message-ID: <87h9755fxz.fsf@handshake.de>

Pedro Franco de Carvalho <reddopp at gmail.com> writes:

> Assume a python program sets a handler function for the
> `signal.SIGALRM` signal, and then schedules an alarm in T seconds with
> `signal.alarm(T)`.
>
> Assume the program later cancels any scheduled alarm with `signal.alarm(0)`.

I do not think so. You might have a "race condition".

I think of the following (hypothetical) situation: the alarm
arrives while "signal.alarm(0)" processing has started but not yet finished.
In this case, the alarm arrival will schedule the alarm handler
activation - but, as "signal.alarm" is "C" implemented, the handler
will not be activated immediately (but delayed until the next
execution of Python code).



From jobmattcon at gmail.com  Fri Nov 18 04:01:01 2016
From: jobmattcon at gmail.com (meInvent bbird)
Date: Fri, 18 Nov 2016 01:01:01 -0800 (PST)
Subject: how to simulate the situation in DNA evolution for finding the
 minimum population needed and minimum samples selected to mating in order to
 no extinction in any one of original species and new species
In-Reply-To: <156581c9-1684-462c-a7d9-0692c6083278@googlegroups.com>
References: <156581c9-1684-462c-a7d9-0692c6083278@googlegroups.com>
Message-ID: <adf56e7e-d96b-4a24-b104-b87a4e60b7ed@googlegroups.com>

i noticed the faces of human repeated or similar, 
and would like to prove whether evolution a several generations will 
return to the original intelligence of ancester

On Friday, November 18, 2016 at 1:55:31 PM UTC+8, meInvent bbird wrote:
> how to simulate the situation in DNA evolution for finding the minimum population needed and minimum samples selected to mating in order to no extinction in any one of original species and new species
> 
> assume mating are randomly selected to become a couple,
> how to keep species good which means no extinction in original species
> 
> i use i+j to get new species, 
> but i do not know whether there is limit in evolution, i assume
> limit is 7, then extra evolution will go back to past species and 
> cycle again
> 
> import matplotlib.pyplot as plt
> import random
> dict = {}
> dist = {}
> maxnum = 5
> allowedmax = 7
> for i in range(1,allowedmax+1):
>     dist[str(i)] = 0
> 
> rr = range (1,maxnum)
> for i in range (1,maxnum):
>     for j in range (1,maxnum):
>         if i < j:
>             print("(" +str(i) + "," + str(j) + ")");
>             dict[str(i) + str(j)] = 1;
>             dist[str(i+j)] = dist[str(i+j)] + 1
>             if i+j > max(rr or [0]):
>                  rr = rr + [i+j];
> 
> original = rr;
> for numberofevolutions in range(1,10):
>     rr2 = []
>     samples = random.sample(original, len(original)-2)
>     print("total rr")
>     print(str(rr))
>     print("samples")
>     print(str(samples))
>     for i in samples:
>         for j in samples:
>             if i < j:
>                 print("(" +str(i) + "," + str(j) + ")");
>                 if i+j > allowedmax:
>                     dict[str(i) + str(j)] = (i+j) % allowedmax;
>                     dist[str((i+j) % allowedmax)] = dist[str((i+j) % allowedmax)] + 1
>                     if ((i+j) % allowedmax) > max(rr2 or [0]):
>                         rr2 = rr2 + [((i+j) % allowedmax)];
>                 else:
>                     dict[str(i) + str(j)] = i+j;
>                     dist[str(i+j)] = dist[str(i+j)] + 1
>                     if i+j > max(rr2 or [0]):
>                         rr2 = rr2 + [i+j];
>     temp = rr
>     rr = rr2
>     rr2 = temp
> 
> plt.bar(range(len(dist)), dist.values(), align='center')
> plt.xticks(range(len(dist)), dist.keys())
> plt.show()



From rosuav at gmail.com  Fri Nov 18 04:03:31 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 18 Nov 2016 20:03:31 +1100
Subject: Can signal.alarm be safely cleared in python?
In-Reply-To: <87h9755fxz.fsf@handshake.de>
References: <CAOfc1cHiV+cTmwAZkCbGGZ0X4g-BoK8nLS7trPHHuLSaN6juag@mail.gmail.com>
 <87h9755fxz.fsf@handshake.de>
Message-ID: <CAPTjJmq2xuOCafGLqB+7WxpYUGD=VLSxbNSdn+i5pvd_UGtAhA@mail.gmail.com>

On Fri, Nov 18, 2016 at 7:51 PM, dieter <dieter at handshake.de> wrote:
> Pedro Franco de Carvalho <reddopp at gmail.com> writes:
>
>> Assume a python program sets a handler function for the
>> `signal.SIGALRM` signal, and then schedules an alarm in T seconds with
>> `signal.alarm(T)`.
>>
>> Assume the program later cancels any scheduled alarm with `signal.alarm(0)`.
>
> I do not think so. You might have a "race condition".
>
> I think of the following (hypothetical) situation: the alarm
> arrives while "signal.alarm(0)" processing has started but not yet finished.
> In this case, the alarm arrival will schedule the alarm handler
> activation - but, as "signal.alarm" is "C" implemented, the handler
> will not be activated immediately (but delayed until the next
> execution of Python code).

That situation would be safe if the cancelling of the timer is in a
sense of "okay, don't bother" rather than "this must now not happen".
For example, you could set a timeout on some operation (say, a
subprocess execution) by setting an alarm. When the process
terminates, you cancel the alarm. If, just after you cancel the alarm,
the signal comes through, that's fine - the handler will already need
to cope with the process having just finished.

So I would say that it's safe IFF you accept this restriction.

ChrisA


From i.venditti at i-write.biz  Fri Nov 18 06:40:21 2016
From: i.venditti at i-write.biz (Irene Venditti)
Date: Fri, 18 Nov 2016 12:40:21 +0100
Subject: Encountering fatal error x80070643 while installing Python
Message-ID: <000001d24190$8c3252d0$a496f870$@i-write.biz>

Hi everybody,

 

I've got a problem with the installation of Python. I am a translator and
currently I'm translating a book on programming Minecraft with Python, from
English to Dutch. 

 

My computer is a Windows 10 computer, 64-bits  (Toshiba Qosmio all in one).

I already had a 2.7 version of Python installed to C:\Program Files
(x86)\Python and stupidly forgot to uninstall this version when I downloaded
and installed version 3.5.1 (required for the book translation).

This didn't seem to be a problem, since version 3.5.1 installed to my
C:\Users\username\appdata\local\... directory. But when I tried to install
Python 3.5.2 and uninstalled both previous versions, the problems began.

Now I cannot install any version of Python anymore, not 3.5.0 or any later
versions. At the end of the procedure I get a fatal error message, with code
0x80070643, saying an error was encountered and the installation couldn't be
completed.

 

What do I do now? I desperately need my Python program for my book
translation.

 

Thanks, Irene

 

Kind regards,

Irene Venditti

i-write translations and texts

 

 <mailto:info at i-write.biz> info at i-write.biz

 <http://www.i-write.biz/> www.i-write.biz

0031 (0)6 220 760 73

 



 



From luchomarzulli at gmail.com  Fri Nov 18 07:46:46 2016
From: luchomarzulli at gmail.com (Luis Marzulli)
Date: Fri, 18 Nov 2016 04:46:46 -0800 (PST)
Subject: IDLEX  association
Message-ID: <d940d611-15a7-4007-a8db-ad8f21ce4393@googlegroups.com>

Hi

When I double click a .py file, a windows appears and immediately disappears.
How can I associate the .py file extension to the IDLEX EDITOR?

Thanks  


From luchomarzulli at gmail.com  Fri Nov 18 07:57:22 2016
From: luchomarzulli at gmail.com (Luis Marzulli)
Date: Fri, 18 Nov 2016 04:57:22 -0800 (PST)
Subject: Style().configure don't works for all widgets
Message-ID: <599f55a2-611f-44fa-a01a-fcb6aa676a4d@googlegroups.com>

Hi
Why 

ttk.Style().configure(".", font=('Courier New', 30, "bold"))

works for Button and Label widgets (and maybe others) and don't works for Entry widget?

Example in Python 3:
from tkinter import *
from tkinter import ttk
from tkinter import font

root = Tk()
ttk.Style().configure(".", font=('Courier New', 30, "bold"))

ttk.Entry(root).grid() # don't works
ttk.Label(root, text="my text").grid() # works OK
ttk.Button(root, text="some text").grid() # works OK

root.mainloop()

Thank you 


From twgrops at googlemail.com  Fri Nov 18 09:48:07 2016
From: twgrops at googlemail.com (twgrops at googlemail.com)
Date: Fri, 18 Nov 2016 06:48:07 -0800 (PST)
Subject: need help to get my python image to move around using tkinter
Message-ID: <e7d8379c-e70a-45f9-92b5-0213eff20586@googlegroups.com>

Hi I am new here and to python,

I am currently studying towards my degree in computer science and have to build a program but I have hit a brick wall. I am trying to make an image move around the canvas. I can make a rectangle move using the following:

#test rectangle
id1=canvas.create_rectangle(3,7,3+10,7+10)

# Generate x and y coordinates for 500 timesteps
for t in range(1, 500):

    x1,y1,x2,y2=canvas.coords(id1)

    # If a boundary has been crossed, reverse the direction
    if x1 >= x_max:
         vx = -10.0
    if y1 <= y_min:
         vy = 5.0
    if y2 >= y_max:
         vy = -5.0
    if x1 <= x_min:
         vx = 10.0

    # Reposition the robot
    canvas.coords(id1,x1+vx,y1+vy,x2+vx,y2+vy)
    canvas.update()

    # Pause for 0.1 seconds, then delete the image
    time.sleep(0.1)

However i would like my image/sprite to move around:

objecttank=canvas.create_image(950,650,image=gif6, anchor= NW)

from what i gather it is because an image only has two axis, x and y but the rectangle seems to have 4 values, my error code keeps saying expecting 4 but got 2. 

Can anyone help me in the right direction, many thanks tom.


From info at egenix.com  Fri Nov 18 10:57:25 2016
From: info at egenix.com (eGenix Team: M.-A. Lemburg)
Date: Fri, 18 Nov 2016 16:57:25 +0100
Subject: ANN: eGenix PyRun - One file Python Runtime 2.2.2
Message-ID: <582F24E5.5060203@egenix.com>

________________________________________________________________________

ANNOUNCING

		eGenix PyRun - One file Python Runtime

			    Version 2.2.2


       An easy-to-use single file relocatable Python run-time -
	  available for Linux, Mac OS X and Unix platforms,
	      with support for Python 2.6, 2.7, 3.4 and
			* now for Python 3.5 *


This announcement is also available on our web-site for online reading:
http://www.egenix.com/company/news/eGenix-PyRun-2.2.2-GA.html

________________________________________________________________________

INTRODUCTION

eGenix PyRun is our open source, one file, no installation version of
Python, making the distribution of a Python interpreter to run based
scripts and applications to Unix based systems as simple as copying a
single file.

eGenix PyRun's executable only needs 11MB for Python 2 and 13MB for
Python 3, but still supports most Python application and scripts - and
it can be compressed to just 3-4MB using upx, if needed.

Compared to a regular Python installation of typically 100MB on disk,
eGenix PyRun is ideal for applications and scripts that need to be
distributed to several target machines, client installations or
customers.

It makes "installing" Python on a Unix based system as simple as
copying a single file.

eGenix has been using eGenix PyRun internally in the mxODBC Connect
Server product since 2008 with great success and decided to make it
available as a stand-alone open-source product.

We provide both the source archive to build your own eGenix PyRun, as
well as pre-compiled binaries for Linux, FreeBSD and Mac OS X, as 32-
and 64-bit versions. The binaries can be downloaded manually, or you
can let our automatic install script install-pyrun take care of the
installation: ./install-pyrun dir and you're done.

Please see the product page for more details:

    http://www.egenix.com/products/python/PyRun/

________________________________________________________________________

NEWS

This minor level release of eGenix PyRun comes with the following
enhancements:

Enhancements / Changes
----------------------

 * Upgraded PyRun to Python 2.7.12, Python 3.4.5 and Python 3.5.2.

 * Fixed rpath setting to properly include the $ORIGIN marker. Without
   this, the rpath setting doesn't work.

 * Added missing lzma module to default PyRun 3.x installation. Please
   note that this adds an additional dependency on libzma.so.5 for
   PyRun for Python 3.4 and 3.5.

install-pyrun Quick Install Enhancements
---------------------------------------------

eGenix PyRun includes a shell script called install-pyrun, which
greatly simplifies installation of PyRun. It works much like the
virtualenv shell script used for creating new virtual environments
(except that there's nothing virtual about PyRun environments).

    https://downloads.egenix.com/python/install-pyrun

With the script, an eGenix PyRun installation is as simple as running:

    ./install-pyrun targetdir

This will automatically detect the platform, download and install the
right pyrun version into targetdir.

We have updated this script since the last release:

 * Updated install-pyrun to default to eGenix PyRun 2.2.2 and its
   feature set.

For a complete list of changes, please see the eGenix PyRun Changelog:

    http://www.egenix.com/products/python/PyRun/changelog.html

________________________________________________________________________

LICENSE

eGenix PyRun is distributed under the eGenix.com Public License 1.1.0
which is an Open Source license similar to the Python license. You can
use eGenix PyRun in both commercial and non-commercial settings
without fee or charge.

Please see our license page for more details:

    http://www.egenix.com/products/python/PyRun/license.html

The package comes with full source code.

________________________________________________________________________

DOWNLOADS

The download archives and instructions for installing eGenix PyRun can
be found at:

    http://www.egenix.com/products/python/PyRun/

As always, we are providing pre-built binaries for all common
platforms: Windows 32/64-bit, Linux 32/64-bit, FreeBSD 32/64-bit, Mac
OS X 32/64-bit. Source code archives are available for installation on
other platforms, such as Solaris, AIX, HP-UX, etc.

_______________________________________________________________________

SUPPORT

Commercial support for this product is available from eGenix.com.
Please see

    http://www.egenix.com/services/support/

for details about our support offerings.

________________________________________________________________________

MORE INFORMATION

For more information about eGenix PyRun, licensing and download
instructions, please visit our web-site:

    http://www.egenix.com/products/python/PyRun/


About eGenix (http://www.egenix.com/):

    eGenix is a Python software project, consulting and product
    company delivering expert services and professional quality
    products for companies, Python users and developers. We specialize
    in database driven applications, large scale software designs and
    integration.

Enjoy,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 18 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/



From grant.b.edwards at gmail.com  Fri Nov 18 11:16:26 2016
From: grant.b.edwards at gmail.com (Grant Edwards)
Date: Fri, 18 Nov 2016 16:16:26 +0000 (UTC)
Subject: ANN: eGenix PyRun - One file Python Runtime 2.2.2
References: <582F24E5.5060203@egenix.com>
Message-ID: <o0n9gq$784$1@blaine.gmane.org>

On 2016-11-18, eGenix Team: M.-A. Lemburg <info at egenix.com> wrote:

> eGenix PyRun is our open source, one file, no installation version of
> Python, making the distribution of a Python interpreter to run based
> scripts and applications

What's a "based" script?

-- 
Grant Edwards               grant.b.edwards        Yow! Those people look
                                  at               exactly like Donnie and
                              gmail.com            Marie Osmond!!



From __peter__ at web.de  Fri Nov 18 11:51:59 2016
From: __peter__ at web.de (Peter Otten)
Date: Fri, 18 Nov 2016 17:51:59 +0100
Subject: Style().configure don't works for all widgets
References: <599f55a2-611f-44fa-a01a-fcb6aa676a4d@googlegroups.com>
Message-ID: <o0nbje$2go$1@blaine.gmane.org>

Luis Marzulli wrote:

> ttk.Style().configure(".", font=('Courier New', 30, "bold"))
> 
> works for Button and Label widgets (and maybe others) and don't works for
> Entry widget?
> 
> Example in Python 3:
> from tkinter import *
> from tkinter import ttk
> from tkinter import font
> 
> root = Tk()
> ttk.Style().configure(".", font=('Courier New', 30, "bold"))
> 
> ttk.Entry(root).grid() # don't works
> ttk.Label(root, text="my text").grid() # works OK
> ttk.Button(root, text="some text").grid() # works OK
> 
> root.mainloop()

I can confirm this, but I have no idea why this is so.
As a workaround you can set the font directly:

FONT = ("Courier New", 30, "bold")
ttk.Style().configure(".", font=FONT)
...
ttk.Entry(root, font=FONT)



From python at mrabarnett.plus.com  Fri Nov 18 12:55:10 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 18 Nov 2016 17:55:10 +0000
Subject: Encountering fatal error x80070643 while installing Python
In-Reply-To: <000001d24190$8c3252d0$a496f870$@i-write.biz>
References: <000001d24190$8c3252d0$a496f870$@i-write.biz>
Message-ID: <5766431f-6ee8-49a1-6521-3bfd742896e9@mrabarnett.plus.com>

On 2016-11-18 11:40, Irene Venditti wrote:
> Hi everybody,
>
> I've got a problem with the installation of Python. I am a translator and
> currently I'm translating a book on programming Minecraft with Python, from
> English to Dutch.
>
> My computer is a Windows 10 computer, 64-bits  (Toshiba Qosmio all in one).
>
> I already had a 2.7 version of Python installed to C:\Program Files
> (x86)\Python and stupidly forgot to uninstall this version when I downloaded
> and installed version 3.5.1 (required for the book translation).
>
> This didn't seem to be a problem, since version 3.5.1 installed to my
> C:\Users\username\appdata\local\... directory. But when I tried to install
> Python 3.5.2 and uninstalled both previous versions, the problems began.
>
> Now I cannot install any version of Python anymore, not 3.5.0 or any later
> versions. At the end of the procedure I get a fatal error message, with code
> 0x80070643, saying an error was encountered and the installation couldn't be
> completed.
>
> What do I do now? I desperately need my Python program for my book
> translation.
>
Have you tried manually deleting those folders? You could just rename 
them instead and delay deleting them until the problem is fixed.

It's recommended to give the Python folder a name that reflects its 
version, e.g. Python35. If you want both the 64-bit and 32-bit versions, 
then Python35-32 and Python35 (or Python35-64).



From mike.reider at gmail.com  Fri Nov 18 13:23:04 2016
From: mike.reider at gmail.com (mike.reider at gmail.com)
Date: Fri, 18 Nov 2016 10:23:04 -0800 (PST)
Subject: Parsing a single-level JSON file
Message-ID: <b9f6b419-7923-4c51-ba0d-c3bed68b0bf2@googlegroups.com>

hi all, 

Im reading in a JSON file that looks like this


[  
   {  
      "name":"myField1",
      "searchable":true,
      "navigable":true,
      "custom":true,
      "clauseNames":[  
         "cf[10190]",
         "Log Details"
      ],
      "orderable":true,
      "id":"customfield_10190",
      "schema":{  
         "customId":10190,
         "type":"string",
         "custom":"com.atlassian.jira.plugin.system.customfieldtypes:textarea"
      }
   },
   {  
      "name":"myField2",
      "searchable":true,
      "navigable":true,
      "custom":true,
      "clauseNames":[  
         "cf[10072]",
         "Sellside Onboarding Checklist"
      ],
      "orderable":true,
      "id":"customfield_10072",
      "schema":{  
         "items":"option",
         "customId":10072,
         "type":"array",
         "custom":"com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes"
      }
   }]


Lets say I want to get the ID # of MyField1, how can I parse this with json lib? Theyre all on the same level, not sure how to target it to go to MyField1 and get "id" value. 

Thanks


From gordon at panix.com  Fri Nov 18 13:33:09 2016
From: gordon at panix.com (John Gordon)
Date: Fri, 18 Nov 2016 18:33:09 +0000 (UTC)
Subject: Parsing a single-level JSON file
References: <b9f6b419-7923-4c51-ba0d-c3bed68b0bf2@googlegroups.com>
Message-ID: <o0nhh5$i5q$1@reader2.panix.com>

In <b9f6b419-7923-4c51-ba0d-c3bed68b0bf2 at googlegroups.com> mike.reider at gmail.com writes:

> Im reading in a JSON file that looks like this
> ... snip ...
> Lets say I want to get the ID # of MyField1, how can I parse this with
> json lib? Theyre all on the same level, not sure how to target it to go
> to MyField1 and get "id" value. 

That data looks like a list of dictionaries:

    import json

    with open("json.dat", "r") as fp:
        data = json.load(fp)
        for item in data:
            if item['name'] == 'myField2':
                print item['id']

-- 
John Gordon                   A is for Amy, who fell down the stairs
gordon at panix.com              B is for Basil, assaulted by bears
                                -- Edward Gorey, "The Gashlycrumb Tinies"



From gordon at panix.com  Fri Nov 18 13:34:51 2016
From: gordon at panix.com (John Gordon)
Date: Fri, 18 Nov 2016 18:34:51 +0000 (UTC)
Subject: Parsing a single-level JSON file
References: <b9f6b419-7923-4c51-ba0d-c3bed68b0bf2@googlegroups.com>
 <o0nhh5$i5q$1@reader2.panix.com>
Message-ID: <o0nhkb$9p3$1@reader2.panix.com>

In <o0nhh5$i5q$1 at reader2.panix.com> John Gordon <gordon at panix.com> writes:

> In <b9f6b419-7923-4c51-ba0d-c3bed68b0bf2 at googlegroups.com> mike.reider at gmail.com writes:

>     with open("json.dat", "r") as fp:
>         data = json.load(fp)
>         for item in data:
>             if item['name'] == 'myField2':

Oops, that should be 'myField1' of course.

-- 
John Gordon                   A is for Amy, who fell down the stairs
gordon at panix.com              B is for Basil, assaulted by bears
                                -- Edward Gorey, "The Gashlycrumb Tinies"



From __peter__ at web.de  Fri Nov 18 13:37:45 2016
From: __peter__ at web.de (Peter Otten)
Date: Fri, 18 Nov 2016 19:37:45 +0100
Subject: Parsing a single-level JSON file
References: <b9f6b419-7923-4c51-ba0d-c3bed68b0bf2@googlegroups.com>
Message-ID: <o0nhpn$tms$1@blaine.gmane.org>

mike.reider at gmail.com wrote:

> hi all,
> 
> Im reading in a JSON file that looks like this
> 
> 
> [  
>    {  
>       "name":"myField1",
>       "searchable":true,
>       "navigable":true,
>       "custom":true,
>       "clauseNames":[
>          "cf[10190]",
>          "Log Details"
>       ],
>       "orderable":true,
>       "id":"customfield_10190",
>       "schema":{
>          "customId":10190,
>          "type":"string",
>          
"custom":"com.atlassian.jira.plugin.system.customfieldtypes:textarea"
>       }
>    },
>    {  
>       "name":"myField2",
>       "searchable":true,
>       "navigable":true,
>       "custom":true,
>       "clauseNames":[
>          "cf[10072]",
>          "Sellside Onboarding Checklist"
>       ],
>       "orderable":true,
>       "id":"customfield_10072",
>       "schema":{
>          "items":"option",
>          "customId":10072,
>          "type":"array",
>          
"custom":"com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes"
>       }
>    }]
> 
> 
> Lets say I want to get the ID # of MyField1, how can I parse this with
> json lib? Theyre all on the same level, not sure how to target it to go to
> MyField1 and get "id" value.

>>> import json
>>> data = json.loads(s)
>>> [wanted] = [d["id"] for d in data if d["name"] == "myField1"]
>>> wanted
'customfield_10190'

If you need to do this often for the same data you can speed up the process 
with a lookup table:

>>> lookup = {d["name"]: d["id"] for d in data}
>>> lookup["myField1"]
'customfield_10190'
>>> lookup["myField2"]
'customfield_10072'

I'm assuming that the names are unique.



From python at mrabarnett.plus.com  Fri Nov 18 13:38:56 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 18 Nov 2016 18:38:56 +0000
Subject: Parsing a single-level JSON file
In-Reply-To: <b9f6b419-7923-4c51-ba0d-c3bed68b0bf2@googlegroups.com>
References: <b9f6b419-7923-4c51-ba0d-c3bed68b0bf2@googlegroups.com>
Message-ID: <149827bb-fac8-b8f6-714f-9339e016d1f0@mrabarnett.plus.com>

On 2016-11-18 18:23, mike.reider at gmail.com wrote:
> hi all,
>
> Im reading in a JSON file that looks like this
>
>
> [
>    {
>       "name":"myField1",
>       "searchable":true,
>       "navigable":true,
>       "custom":true,
>       "clauseNames":[
>          "cf[10190]",
>          "Log Details"
>       ],
>       "orderable":true,
>       "id":"customfield_10190",
>       "schema":{
>          "customId":10190,
>          "type":"string",
>          "custom":"com.atlassian.jira.plugin.system.customfieldtypes:textarea"
>       }
>    },
>    {
>       "name":"myField2",
>       "searchable":true,
>       "navigable":true,
>       "custom":true,
>       "clauseNames":[
>          "cf[10072]",
>          "Sellside Onboarding Checklist"
>       ],
>       "orderable":true,
>       "id":"customfield_10072",
>       "schema":{
>          "items":"option",
>          "customId":10072,
>          "type":"array",
>          "custom":"com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes"
>       }
>    }]
>
>
> Lets say I want to get the ID # of MyField1, how can I parse this with json lib? Theyre all on the same level, not sure how to target it to go to MyField1 and get "id" value.
>
> Thanks
>
Parsing it is easy:

with open(json_path) as json_file:
     data_list = json.load(json_file)


If it's a one-off lookup, just do a linear search:

for entry in data_list:
     if entry['name'] == 'myField1':
         print('ID is {}'.format(entry['id']))
         break
else:
     print('Not found!')


If you're going to do multiple lookups, turn it into a dict, using the 
'name' field as the key:

data_dict = {entry['name']: entry for entry in data_list}
if 'myField1' in data_dict:
     print('ID is {}'.format(data_dict['myField1']['id']))
else:
     print('Not found!')



From __peter__ at web.de  Fri Nov 18 14:06:13 2016
From: __peter__ at web.de (Peter Otten)
Date: Fri, 18 Nov 2016 20:06:13 +0100
Subject: need help to get my python image to move around using tkinter
References: <e7d8379c-e70a-45f9-92b5-0213eff20586@googlegroups.com>
Message-ID: <o0njf4$781$1@blaine.gmane.org>

twgrops--- via Python-list wrote:

> Hi I am new here and to python,
> 
> I am currently studying towards my degree in computer science and have to
> build a program but I have hit a brick wall. I am trying to make an image
> move around the canvas. I can make a rectangle move using the following:
> 
> #test rectangle
> id1=canvas.create_rectangle(3,7,3+10,7+10)
> 
> # Generate x and y coordinates for 500 timesteps
> for t in range(1, 500):
> 
>     x1,y1,x2,y2=canvas.coords(id1)
> 
>     # If a boundary has been crossed, reverse the direction
>     if x1 >= x_max:
>          vx = -10.0
>     if y1 <= y_min:
>          vy = 5.0
>     if y2 >= y_max:
>          vy = -5.0
>     if x1 <= x_min:
>          vx = 10.0
> 
>     # Reposition the robot
>     canvas.coords(id1,x1+vx,y1+vy,x2+vx,y2+vy)
>     canvas.update()
> 
>     # Pause for 0.1 seconds, then delete the image
>     time.sleep(0.1)

You should never use sleep() in conjunction with tkinter. Instead make a 
little function that schedules itself for invocation after a few 
milliseconds:

def move():
   ... # move sprits
   root.after(100, move) # have tkinter call move() 
                         # again after 100 milliseconds

That way the event loop is undisturbed and the application can respond to 
user input.

> However i would like my image/sprite to move around:
> 
> objecttank=canvas.create_image(950,650,image=gif6, anchor= NW)
> 
> from what i gather it is because an image only has two axis, x and y but
> the rectangle seems to have 4 values, my error code keeps saying expecting
> 4 but got 2.

Always provide the exact code causing the error, and the traceback it 
generated. Use copy and paste instead of paraphrasing.
 
> Can anyone help me in the right direction, many thanks tom.

I think I have done this before, but cannot find it atm, so instead of a 
link here's a new demo:

try:
    import tkinter as tk
except ImportError:
    import Tkinter as tk

import random

WIDTH = 1024
HEIGHT = 768


def random_speed():
    return random.choice([-1, 1]) * random.randrange(5, 15)


def random_image(canvas, image):
    """Place `image` on `canvas` with random pos and speed.
    """
    return Image(
        canvas,
        image,
        random.randrange(WIDTH-image.width()),
        random.randrange(HEIGHT-image.height()),
        random_speed(),
        random_speed(),
    )


class Image(object):
    def __init__(self, canvas, image, x, y, vx, vy):
        self.canvas = canvas
        self.image = image
        self.x = x
        self.y = y
        self.vx = vx
        self.vy = vy
        self.w = image.width()
        self.h = image.height()
        self.id = canvas.create_image(x, y, image=image, anchor=tk.NW)

    @property
    def right(self):
        return self.x + self.w

    @property
    def bottom(self):
        return self.y + self.h

    def move(self):
        self.x += self.vx
        self.y += self.vy
        self.canvas.move(self.id, self.vx, self.vy)
        if self.bottom > HEIGHT or self.y < 0:
            self.vy = - self.vy
        if self.right > WIDTH or self.x < 0:
            self.vx = - self.vx


def move():
    for image in images:
        image.move()
    root.after(50, move)


if __name__ == "__main__":
    root = tk.Tk()
    ball = tk.PhotoImage(file="ball.gif")
    canvas = tk.Canvas(root, width=WIDTH, height=HEIGHT)
    canvas.pack()
    images = [random_image(canvas, ball) for _ in range(10)]
    move()
    root.mainloop()




From mike.reider at gmail.com  Fri Nov 18 14:23:43 2016
From: mike.reider at gmail.com (mike.reider at gmail.com)
Date: Fri, 18 Nov 2016 11:23:43 -0800 (PST)
Subject: Parsing a single-level JSON file
In-Reply-To: <b9f6b419-7923-4c51-ba0d-c3bed68b0bf2@googlegroups.com>
References: <b9f6b419-7923-4c51-ba0d-c3bed68b0bf2@googlegroups.com>
Message-ID: <ffb60501-bd4a-4771-9d16-b323a8055e84@googlegroups.com>

On Friday, November 18, 2016 at 1:23:18 PM UTC-5, mike.... at gmail.com wrote:
> hi all, 
> 
> Im reading in a JSON file that looks like this
> 
> 
> [  
>    {  
>       "name":"myField1",
>       "searchable":true,
>       "navigable":true,
>       "custom":true,
>       "clauseNames":[  
>          "cf[10190]",
>          "Log Details"
>       ],
>       "orderable":true,
>       "id":"customfield_10190",
>       "schema":{  
>          "customId":10190,
>          "type":"string",
>          "custom":"com.atlassian.jira.plugin.system.customfieldtypes:textarea"
>       }
>    },
>    {  
>       "name":"myField2",
>       "searchable":true,
>       "navigable":true,
>       "custom":true,
>       "clauseNames":[  
>          "cf[10072]",
>          "Sellside Onboarding Checklist"
>       ],
>       "orderable":true,
>       "id":"customfield_10072",
>       "schema":{  
>          "items":"option",
>          "customId":10072,
>          "type":"array",
>          "custom":"com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes"
>       }
>    }]
> 
> 
> Lets say I want to get the ID # of MyField1, how can I parse this with json lib? Theyre all on the same level, not sure how to target it to go to MyField1 and get "id" value. 
> 
> Thanks



thanks everyone for the feedback, i got it to work like this using multi-dim dictionary

# get JSON to parse
url = "https://"+jira_server+"/rest/api/2/field"
req = requests.get(url,auth=(jira_user,jira_pw), verify=False)
jsonfile = req.json()

# save as key,val pair file
cf = {}
for item in jsonfile:
    name = item.get("name")
    fid =  item.get("id")
    cf[name] = { 'id' : fid }
         
with open(base_dir+'/fields.json','w') as f:
    json.dump(cf,f)



From mike.reider at gmail.com  Fri Nov 18 14:28:30 2016
From: mike.reider at gmail.com (mike.reider at gmail.com)
Date: Fri, 18 Nov 2016 11:28:30 -0800 (PST)
Subject: Parsing a single-level JSON file
In-Reply-To: <b9f6b419-7923-4c51-ba0d-c3bed68b0bf2@googlegroups.com>
References: <b9f6b419-7923-4c51-ba0d-c3bed68b0bf2@googlegroups.com>
Message-ID: <f4c61e38-7f33-4ce5-807b-db159330ffea@googlegroups.com>



the end result file looks like this

cat fields.json

{"myField1": {"id": "customfield_10600"}, "myField2": {"id": "customfield_11334"}, "myField3": {"id": "customfield_993434"}, etc etc


From twgrops at googlemail.com  Fri Nov 18 17:36:16 2016
From: twgrops at googlemail.com (Thomas Grops)
Date: Fri, 18 Nov 2016 14:36:16 -0800 (PST)
Subject: need help to get my python image to move around using tkinter
In-Reply-To: <e7d8379c-e70a-45f9-92b5-0213eff20586@googlegroups.com>
References: <e7d8379c-e70a-45f9-92b5-0213eff20586@googlegroups.com>
Message-ID: <916c47ba-bab9-4e73-90df-6b352fc30f00@googlegroups.com>

thankyou so much, that is the exact help I required to put me in the right direction :D


From eryksun at gmail.com  Fri Nov 18 18:43:44 2016
From: eryksun at gmail.com (eryk sun)
Date: Fri, 18 Nov 2016 23:43:44 +0000
Subject: Encountering fatal error x80070643 while installing Python
In-Reply-To: <5766431f-6ee8-49a1-6521-3bfd742896e9@mrabarnett.plus.com>
References: <000001d24190$8c3252d0$a496f870$@i-write.biz>
 <5766431f-6ee8-49a1-6521-3bfd742896e9@mrabarnett.plus.com>
Message-ID: <CACL+1asJNveaH8dzW2iNdoNPDFJhKwZkTLrSVAjjaiO9Q891Yg@mail.gmail.com>

On Fri, Nov 18, 2016 at 5:55 PM, MRAB <python at mrabarnett.plus.com> wrote:
> On 2016-11-18 11:40, Irene Venditti wrote:
>>
>> This didn't seem to be a problem, since version 3.5.1 installed to my
>> C:\Users\username\appdata\local\... directory. But when I tried to install
>> Python 3.5.2 and uninstalled both previous versions, the problems began.
>>
>> Now I cannot install any version of Python anymore, not 3.5.0 or any later
>> versions. At the end of the procedure I get a fatal error message, with
>> code 0x80070643, saying an error was encountered and the installation
>> couldn't be completed.
>
> Have you tried manually deleting those folders? You could just rename them
> instead and delay deleting them until the problem is fixed.
>
> It's recommended to give the Python folder a name that reflects its version,
> e.g. Python35. If you want both the 64-bit and 32-bit versions, then
> Python35-32 and Python35 (or Python35-64).

It looks like the OP is using the default per-user target directory,
"%LocalAppData%\Programs\Python\Python35[-32]". It's possible
something was left there by the old 3.5.1 installation, such as
pip/setuptools in site-packages, which is causing the 3.5.2
installation to fail. I would first try the "repair" option using the
3.5.1 installer, if that's an available option. Then upgrade to 3.5.2.
You shouldn't need to remove 3.5.1 to install 3.5.2. It's an in-place
upgrade.


From eryksun at gmail.com  Fri Nov 18 19:46:55 2016
From: eryksun at gmail.com (eryk sun)
Date: Sat, 19 Nov 2016 00:46:55 +0000
Subject: IDLEX association
In-Reply-To: <hmju2c5v23497ohrjtdr3jnjmfcuhusrq2@4ax.com>
References: <d940d611-15a7-4007-a8db-ad8f21ce4393@googlegroups.com>
 <hmju2c5v23497ohrjtdr3jnjmfcuhusrq2@4ax.com>
Message-ID: <CACL+1at04zVUDw1NSqz=b5qLcM63MS_J9BbwLWyFhETy8Yqw4g@mail.gmail.com>

On Fri, Nov 18, 2016 at 7:09 PM, Dennis Lee Bieber
<wlfraed at ix.netcom.com> wrote:
> On Fri, 18 Nov 2016 04:46:46 -0800 (PST), Luis Marzulli
> <luchomarzulli at gmail.com> declaimed the following:
>
>>When I double click a .py file, a windows appears and immediately disappears.
>>How can I associate the .py file extension to the IDLEX EDITOR?
>
>         You really don't want to do that (I'm not familiar with "IDLEX", but
> Python's IDLE is itself a Python script, so if Python scripts were
> associated with the editor script, you'd get a recursion of the editor
> trying to open itself in the editor).

No, the file association would run IDLE[X] via pyw.exe or pythonw.exe.

If Python is installed for all users and you have the default
associations selected as you user choice in the shell (i.e.
Python.File and Python.NoConFile), then you can easily modify the .py
and .pyw file association using an elevated command prompt.
(Ironically it's harder to modify this for a per-user installation,
since cmd's assoc and ftype commands use only the system registry
hive.) For example:

Show the current association:

    C:\>assoc .py
    .py=Python.File

    C:\>ftype Python.File
    Python.File="C:\Windows\py.exe" "%1" %*

(Note that assoc shows the system association, but your user choice in
the shell may be different. This choice is buried in the registry
under HKCU\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\FileExts\.py\UserChoice.)

Modify the "Python.File" ProgId to run IDLE 3.5 (running IDLEX is
probably similar, but I don't use it):

    C:\>ftype Python.File="C:\Windows\pyw.exe" -3.5 -m idlelib "%1" %*
    Python.File="C:\Windows\pyw.exe" -3.5 -m idlelib "%1" %*

You can easily restore the association back to the default:

    C:\>ftype Python.File="C:\Windows\py.exe" "%1" %*
    Python.File="C:\Windows\py.exe" "%1" %*

and run a test script:

    C:\>type test.py
    #!/usr/bin/python3
    import sys
    print(sys.executable)

    C:\>test.py
    C:\Program Files\Python36\python.exe

Personally I don't use an editor as the default action for .py files.
Instead I add a right-click "edit" action. For the command line, I
have an edit script that invokes this action on a file.


From jfong at ms4.hinet.net  Fri Nov 18 20:44:53 2016
From: jfong at ms4.hinet.net (jfong at ms4.hinet.net)
Date: Fri, 18 Nov 2016 17:44:53 -0800 (PST)
Subject: How to append a modified list into a list?
Message-ID: <cbba816d-8bfd-4578-a58d-d12d33a9ff69@googlegroups.com>

I have a working list 'tbl' and recording list 'm'. I want to append 'tbl' into 'm' each time when the 'tbl' was modified. I will record the change by append it through the function 'apl'.

For example:

>>>tbl=[0,0]
>>>m=[]

>>>tbl[0]=1
>>>apl(tbl)
>>>m
[[1,0]]

>>>tbl[1]=2
>>>apl(tbl)
>>>m
[[1,0], [1,2]]

How to define this function properly?

Obviously the most intuitive way doesn't work.
def apl0(tbl):
    m.append(tbl)

and introducing a local variable will not help either.
def apl1(tbl):
    w=tbl
    m.append(w)

I figure out a workable way, but looks ugly.
def apl2(tbl):
    w=[]
    w[:]=tbl
    m.append(w)

I know those binding tricks between names and objects. Just wondering if there is an elegant way of doing this:-)

--Jach


From limetree377 at gmail.com  Fri Nov 18 20:51:34 2016
From: limetree377 at gmail.com (limetree377 at gmail.com)
Date: Fri, 18 Nov 2016 17:51:34 -0800 (PST)
Subject: how to multiply two matrices with different size?
Message-ID: <82bf5c38-4900-4707-ac47-2b36db3b6579@googlegroups.com>

Hi :)
I'm trying to multiply two matrices that has different size.

-------------------------------------code-------------------------

import numpy as np

a = np.random.randn(4, 3)
b = np.random.randn(4, 1)

print a
print b

-------------------------------------code-------------------------

How should I multiply a and b so that the multipled matrix's size is 4*3?

I want to multiply matrix 'b' to each row of matrix 'a'.
So that if matrix a is
[[1, 2, 3],
  [2, 3, 4]]
and b is
[[2],
[3]]
a*b is
[[2, 4, 6],
  [4, 6 ,8]]

Plz help me!

if possible, plz use numpy


From ian.g.kelly at gmail.com  Fri Nov 18 21:23:51 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Fri, 18 Nov 2016 19:23:51 -0700
Subject: how to multiply two matrices with different size?
In-Reply-To: <82bf5c38-4900-4707-ac47-2b36db3b6579@googlegroups.com>
References: <82bf5c38-4900-4707-ac47-2b36db3b6579@googlegroups.com>
Message-ID: <CALwzidnGC-MyRus10iM4hEAAfWqG4s5fmpzqAwf-=LsWHphoRw@mail.gmail.com>

On Fri, Nov 18, 2016 at 6:51 PM,  <limetree377 at gmail.com> wrote:
> Hi :)
> I'm trying to multiply two matrices that has different size.
>
> -------------------------------------code-------------------------
>
> import numpy as np
>
> a = np.random.randn(4, 3)
> b = np.random.randn(4, 1)
>
> print a
> print b
>
> -------------------------------------code-------------------------
>
> How should I multiply a and b so that the multipled matrix's size is 4*3?
>
> I want to multiply matrix 'b' to each row of matrix 'a'.
> So that if matrix a is
> [[1, 2, 3],
>   [2, 3, 4]]
> and b is
> [[2],
> [3]]
> a*b is
> [[2, 4, 6],
>   [4, 6 ,8]]
>
> Plz help me!
>
> if possible, plz use numpy

You don't need to do anything special.

>>> import numpy as np
>>> a = np.array([[1,2,3],[2,3,4]])
>>> b = np.array([[2], [3]])
>>> a
array([[1, 2, 3],
       [2, 3, 4]])
>>> b
array([[2],
       [3]])
>>> a * b
array([[ 2,  4,  6],
       [ 6,  9, 12]])

(I'm assuming this is actually the output you want. The output you
posted is jut a * 2.)


From ian.g.kelly at gmail.com  Fri Nov 18 21:35:11 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Fri, 18 Nov 2016 19:35:11 -0700
Subject: How to append a modified list into a list?
In-Reply-To: <cbba816d-8bfd-4578-a58d-d12d33a9ff69@googlegroups.com>
References: <cbba816d-8bfd-4578-a58d-d12d33a9ff69@googlegroups.com>
Message-ID: <CALwzidkBw4-779u+HEEEGJw90YbJ8i_zJQ=ae3VoxxTzAMsSpw@mail.gmail.com>

On Nov 18, 2016 6:47 PM, <jfong at ms4.hinet.net> wrote:

I have a working list 'tbl' and recording list 'm'. I want to append 'tbl'
into 'm' each time when the 'tbl' was modified. I will record the change by
append it through the function 'apl'.

For example:

>>>tbl=[0,0]
>>>m=[]

>>>tbl[0]=1
>>>apl(tbl)
>>>m
[[1,0]]

>>>tbl[1]=2
>>>apl(tbl)
>>>m
[[1,0], [1,2]]

How to define this function properly?

Obviously the most intuitive way doesn't work.
def apl0(tbl):
    m.append(tbl)

and introducing a local variable will not help either.
def apl1(tbl):
    w=tbl
    m.append(w)

I figure out a workable way, but looks ugly.
def apl2(tbl):
    w=[]
    w[:]=tbl
    m.append(w)

I know those binding tricks between names and objects. Just wondering if
there is an elegant way of doing this:-)


The important thing is to copy the list, not rebind it.

def apl(tbl):
    m.append(tbl[:])

Or:

def apl(tbl):
    m.append(list(tbl))


From steve+python at pearwood.info  Fri Nov 18 22:01:10 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sat, 19 Nov 2016 14:01:10 +1100
Subject: How to append a modified list into a list?
References: <cbba816d-8bfd-4578-a58d-d12d33a9ff69@googlegroups.com>
Message-ID: <582fc077$0$1599$c3e8da3$5496439d@news.astraweb.com>

On Sat, 19 Nov 2016 12:44 pm, jfong at ms4.hinet.net wrote:

> I have a working list 'tbl' and recording list 'm'. I want to append 'tbl'
> into 'm' each time when the 'tbl' was modified. I will record the change
> by append it through the function 'apl'.
[...]
> Obviously the most intuitive way doesn't work.
> def apl0(tbl):
>     m.append(tbl)

That works perfectly -- it just doesn't do what you want.

It sounds like what you want is to append a COPY of the list rather than the
list itself. I'm not sure why, making copies of things in Python is usually
rare, but perhaps I don't understand what you are doing with this "working
list" and "recording list".

The simplest way to do this is:


m.append(tbl[:])  # use slicing to make a copy


In newer versions of Python, you can do this:


m.append(tbl.copy())


But that won't work in Python 2.7. Or you can do this:



from copy import copy
m.append(copy(tbl))



But the most Pythonic way (the most standard way) is to use a slice to copy
the list: m.append(tbl[:])

 
> and introducing a local variable will not help either.
> def apl1(tbl):
>     w=tbl
>     m.append(w)

Of course not. Assignment does not make a copy.





-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Fri Nov 18 23:34:01 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sat, 19 Nov 2016 15:34:01 +1100
Subject: help on "from deen import *" vs. "import deen"
References: <5d48ae5f-17f1-fca7-1239-d22c2950363c@gmail.com>
 <mailman.154.1479266093.30068.python-list@python.org>
 <c4c12dd3-7d2d-4a87-bdbc-0e752dbf5318@googlegroups.com>
 <ababec33-ad77-60f4-367f-aa89aad22230@gmail.com>
 <mailman.7.1479397099.20410.python-list@python.org>
 <023fdec4-3630-4030-a58b-24028dde4b41@googlegroups.com>
 <5ee9f8ce-e3a6-8163-02da-f5d928d219e1@gmail.com>
 <mailman.24.1479438177.20410.python-list@python.org>
 <90551ab1-c0b9-48db-85c3-c9487926b3f7@googlegroups.com>
 <kaiu2cdjgd8bu94tvhvve532ug3v8r3uf6@4ax.com>
 <mailman.40.1479495082.20410.python-list@python.org>
Message-ID: <582fd63b$0$1598$c3e8da3$5496439d@news.astraweb.com>

On Sat, 19 Nov 2016 05:50 am, Dennis Lee Bieber wrote:

> Python's name binding is very easy to state in English: Assignment in
> Python attaches the name (on the left hand side of the statement) to the
> object (that is the result of the right hand side).

What do you mean "attach"? What's a name? What's an object?


What happens if you do this?

spam = eggs = cheese = obj

Is that different from:

spam = obj
eggs = obj
cheese = obj


or from this?

spam = obj
eggs = spam
cheese = eggs


As an expert Python program with more than a decade's experience, you
probably think these questions are so basic that they are obvious. But to a
beginner, they aren't even close to obvious.


https://www.edutopia.org/blog/the-curse-of-knowledge-chris-reddy

https://signalvnoise.com/posts/213-the-curse-of-knowledge


> In contrast, assignment in C, FORTRAN, BASIC, COBOL can be stated as:
> Assignment in <language> copies the contents (value) of the memory
> location identified by the right hand side to the memory location
> identified by the left hand side.

Well that doesn't help...

foo = 999 + 1;

What memory location is identified by the right hand side?



> The Python actually comes out cleaner -- no need to explain memory
> locations, contents/values, and copying.

No, but you do have to explain "bind", "name" and "value".


> If you need a visual, in Python, fully qualified names are labels with
> a string; the other end of the string is taped to an object.

Analogies are great, but you also have to explain the limits of the analogy.
And analogies can mislead too.


What's the first end attached to? What precisely is a name?


Since we can do this:

# name with string attached to object
fred-------999


and we can do this:

# two names with two strings both attached to the same object
fred--------------999
                   |
george-------------+


can we do any of these? If we can't, why not?


# object with string not attached to any name
      -----------999


# name with string not attached to any object
fred-----------


# name with string attached to another name, attached to object
fred---------george----------999


# object with string attached to name
fred----------999-----------george


# two names, attached to each other, with no object
fred--------+
            |
george------+


# name with two pieces of string attached to two objects
fred-----------------999
   |
   +---------888


# name with string attached to another string?
fred---------+
             |
george-------+------999



# two objects tied together
           999---------888


# name with no string
fred


# object with no string
          999


# string tied in a loop (with or without a name)

fred+-----------+
    |           |
    +-----------+

    +-----------+
    |           |
    +-----------+



These aren't silly questions. Some of them are meaningful, some of them
aren't. They're questions suggested by the "label with string tied to an
object" model of name binding. As experts, we're so used to the limits of
the model that we don't stop to think about all the ways that it can
mislead. But a beginner doesn't know the limits of the model, and doesn't
know how to apply the model to their code.




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From jfong at ms4.hinet.net  Sat Nov 19 00:40:43 2016
From: jfong at ms4.hinet.net (jfong at ms4.hinet.net)
Date: Fri, 18 Nov 2016 21:40:43 -0800 (PST)
Subject: How to append a modified list into a list?
In-Reply-To: <582fc077$0$1599$c3e8da3$5496439d@news.astraweb.com>
References: <cbba816d-8bfd-4578-a58d-d12d33a9ff69@googlegroups.com>
 <582fc077$0$1599$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <6364360c-9645-4110-b0df-17fcbbf89913@googlegroups.com>

Oh, I don't know slice well enough:-(

So, slice tbl[:] will create a new object (a copy of tbl) which can be passed as a function argument
m.append(tbl[:])

or bind to a new name
w=tbl[:]

or re-bind to itself
w[:]=tbl

Thanks you, Ian and Steve.

Steve D'Aprano at 2016/11/19 11:01:26AM wrote:
> In newer versions of Python, you can do this:
> m.append(tbl.copy())

I like it:-)

--Jach



From rosuav at gmail.com  Sat Nov 19 01:58:21 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 19 Nov 2016 17:58:21 +1100
Subject: help on "from deen import *" vs. "import deen"
In-Reply-To: <582fd63b$0$1598$c3e8da3$5496439d@news.astraweb.com>
References: <5d48ae5f-17f1-fca7-1239-d22c2950363c@gmail.com>
 <mailman.154.1479266093.30068.python-list@python.org>
 <c4c12dd3-7d2d-4a87-bdbc-0e752dbf5318@googlegroups.com>
 <ababec33-ad77-60f4-367f-aa89aad22230@gmail.com>
 <mailman.7.1479397099.20410.python-list@python.org>
 <023fdec4-3630-4030-a58b-24028dde4b41@googlegroups.com>
 <5ee9f8ce-e3a6-8163-02da-f5d928d219e1@gmail.com>
 <mailman.24.1479438177.20410.python-list@python.org>
 <90551ab1-c0b9-48db-85c3-c9487926b3f7@googlegroups.com>
 <kaiu2cdjgd8bu94tvhvve532ug3v8r3uf6@4ax.com>
 <mailman.40.1479495082.20410.python-list@python.org>
 <582fd63b$0$1598$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPTjJmroWO8iKnFRrGA_9yjKMfJdiuPpDhM0a3jxZ=QZztJHgA@mail.gmail.com>

On Sat, Nov 19, 2016 at 3:34 PM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> What happens if you do this?
>
> spam = eggs = cheese = obj
>
> Is that different from:
>
> spam = obj
> eggs = obj
> cheese = obj
>
>
> or from this?
>
> spam = obj
> eggs = spam
> cheese = eggs
> ...
> These aren't silly questions.

Indeed, they are not silly. It surprised me (as a C programmer) that
the assignments happen left-to-right, rather than right-to-left (in C,
cheese would be set first, then eggs, then spam). These are questions
filled with subtleties.

ChrisA


From __peter__ at web.de  Sat Nov 19 04:40:06 2016
From: __peter__ at web.de (Peter Otten)
Date: Sat, 19 Nov 2016 10:40:06 +0100
Subject: How to append a modified list into a list?
References: <cbba816d-8bfd-4578-a58d-d12d33a9ff69@googlegroups.com>
Message-ID: <o0p6lj$1eh$1@blaine.gmane.org>

jfong at ms4.hinet.net wrote:

> I have a working list 'tbl' and recording list 'm'. I want to append 'tbl'
> into 'm' each time when the 'tbl' was modified. I will record the change
> by append it through the function 'apl'.
> 
> For example:
> 
>>>>tbl=[0,0]
>>>>m=[]
> 
>>>>tbl[0]=1
>>>>apl(tbl)
>>>>m
> [[1,0]]
> 
>>>>tbl[1]=2
>>>>apl(tbl)
>>>>m
> [[1,0], [1,2]]
> 
> How to define this function properly?
> 
> Obviously the most intuitive way doesn't work.
> def apl0(tbl):
>     m.append(tbl)
> 
> and introducing a local variable will not help either.
> def apl1(tbl):
>     w=tbl
>     m.append(w)
> 
> I figure out a workable way, but looks ugly.
> def apl2(tbl):
>     w=[]
>     w[:]=tbl
>     m.append(w)
> 
> I know those binding tricks between names and objects. Just wondering if
> there is an elegant way of doing this:-)
> 
> --Jach

And now for something completely different ;)

What if you only record the changes to the list? For a long list that would 
save space at the expense of calculation time. For example:

$ cat list_history.py
import collections
from itertools import islice

class List(collections.Sequence):
    def __init__(self, items):
        self._items = list(items)
        self._history = []
    def __setitem__(self, index, value):
        if isinstance(index, slice):
            value = list(value)
            old = self._items[index]
            if len(old) != len(value):
                raise ValueError("size changes not supported")
        else:
            old = self._items[index]
        self._history.append((index, old))
        self._items[index] = value
    def __getitem__(self, index):
        return self._items[index]
    def __len__(self):
        return len(self._items)
    def __repr__(self):
        return repr(self._items)
    def historic_state(self, n):
        items = self._items[:]
        for index, value in islice(reversed(self._history), n):
            items[index] = value
        return items
    def history_len(self):
        return len(self._history) + 1

if __name__ == "__main__":
    items = List("abcdefghijk")
    print(items)
    # apply some changes
    items[3] = "foo"
    items[7] = "bar"
    items[3] = 42
    items[3:6] = "xyz"
    items[5:7] = (1, 2)
    print(items)
    # let's go back in time
    for i in range(items.history_len()):
        print("#{}: {}".format(i, items.historic_state(i)))
$ python3 list_history.py 
['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k']
['a', 'b', 'c', 'x', 'y', 1, 2, 'bar', 'i', 'j', 'k']
#0: ['a', 'b', 'c', 'x', 'y', 1, 2, 'bar', 'i', 'j', 'k']
#1: ['a', 'b', 'c', 'x', 'y', 'z', 'g', 'bar', 'i', 'j', 'k']
#2: ['a', 'b', 'c', 42, 'e', 'f', 'g', 'bar', 'i', 'j', 'k']
#3: ['a', 'b', 'c', 'foo', 'e', 'f', 'g', 'bar', 'i', 'j', 'k']
#4: ['a', 'b', 'c', 'foo', 'e', 'f', 'g', 'h', 'i', 'j', 'k']
#5: ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k']




From jfong at ms4.hinet.net  Sat Nov 19 04:49:53 2016
From: jfong at ms4.hinet.net (jfong at ms4.hinet.net)
Date: Sat, 19 Nov 2016 01:49:53 -0800 (PST)
Subject: help on "from deen import *" vs. "import deen"
In-Reply-To: <mailman.48.1479538706.20410.python-list@python.org>
References: <5d48ae5f-17f1-fca7-1239-d22c2950363c@gmail.com>
 <mailman.154.1479266093.30068.python-list@python.org>
 <c4c12dd3-7d2d-4a87-bdbc-0e752dbf5318@googlegroups.com>
 <ababec33-ad77-60f4-367f-aa89aad22230@gmail.com>
 <mailman.7.1479397099.20410.python-list@python.org>
 <023fdec4-3630-4030-a58b-24028dde4b41@googlegroups.com>
 <5ee9f8ce-e3a6-8163-02da-f5d928d219e1@gmail.com>
 <mailman.24.1479438177.20410.python-list@python.org>
 <90551ab1-c0b9-48db-85c3-c9487926b3f7@googlegroups.com>
 <kaiu2cdjgd8bu94tvhvve532ug3v8r3uf6@4ax.com>
 <mailman.40.1479495082.20410.python-list@python.org>
 <582fd63b$0$1598$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmroWO8iKnFRrGA_9yjKMfJdiuPpDhM0a3jxZ=QZztJHgA@mail.gmail.com>
 <mailman.48.1479538706.20410.python-list@python.org>
Message-ID: <67c8a87a-4af9-440b-a5b0-b9903a54ad09@googlegroups.com>

Chris Angelico at 2016/11/19 2:58:41PM wrote:
> On Sat, Nov 19, 2016 at 3:34 PM, Steve D'Aprano
> <steve+python at pearwood.info> wrote:
> > What happens if you do this?
> >
> > spam = eggs = cheese = obj
> >
> > Is that different from:
> >
> > spam = obj
> > eggs = obj
> > cheese = obj
> >
> >
> > or from this?
> >
> > spam = obj
> > eggs = spam
> > cheese = eggs
> > ...
> > These aren't silly questions.
> 
> Indeed, they are not silly. It surprised me (as a C programmer) that
> the assignments happen left-to-right, rather than right-to-left (in C,
> cheese would be set first, then eggs, then spam). These are questions
> filled with subtleties.
> 
> ChrisA

Why this conversation makes me feel like a fool? Are they different?

--Jach


From porton at narod.ru  Sat Nov 19 10:00:09 2016
From: porton at narod.ru (Victor Porton)
Date: Sat, 19 Nov 2016 17:00:09 +0200
Subject: Two variants of class hierachy
Message-ID: <o0ppdo$bpu$1@gioia.aioe.org>

I am developing software which shows hierarchical information (tree), 
including issues and comments from BitBucket (comments are sub-nodes of 
issues, thus it forms a tree).

There are two kinds of objects in the hierarchy: a. with a (possibly long) 
paginated list of childs; b. with a short list of strings, each string being 
associated with a child object.

I have two variants of class inheritance in mind. Please help to decide 
which is better.	

The first one declares only one base class, but some its method remain 
unimplemented (raise NotImplementedError) even in derived classes.

The second one defines two distinct base classes 
HierarchyLevelWithPagination (for objects of above described class "a") and 
HierarchyLevelWithShortList (for objects of above described class "b"), but 
use multiple inheritance.

# VARIANT 1: #

class HierarchyLevel(object):
    def get_node(self):
        return None

    def childs(self, url, page, per_page=None):
        raise NotImplementedError()

    def short_childs(self):
        raise NotImplementedError()

class BitBucketHierarchyLevel(HierarchyLevel):
    ...

# A implements only childs() but not short_childs()
class A(BitBucketHierarchyLevel):
    ...

# B implements only short_childs() but not childs()
class B(BitBucketHierarchyLevel):
    ...

## OR ##
# VARIANT 2: #

class HierarchyLevel(object):
    def get_node(self):
        return None

class HierarchyLevelWithPagination(HierarchyLevel):
    def childs(self, url, page, per_page=None):
        raise NotImplementedError()

class HierarchyLevelWithShortList(HierarchyLevel):
    def short_childs(self):
        raise NotImplementedError()

## THEN ##

# code specific for BitBucket
class BitBucketHierarchyLevel(HierarchyLevel):
    ...

# diamonds:

class A(BitBucketHierarchyLevel, HierarchyLevelWithPagination):
    ...

class B(BitBucketHierarchyLevel, HierarchyLevelWithShortList):
    ...

-- 
Victor Porton - http://portonvictor.org


From steve+python at pearwood.info  Sat Nov 19 10:46:41 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sun, 20 Nov 2016 02:46:41 +1100
Subject: help on "from deen import *" vs. "import deen"
References: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>
 <bb816fbf-1df7-522d-bab7-9f7a65deaef3@mrabarnett.plus.com>
 <mailman.147.1479180690.30068.python-list@python.org>
 <1a6f991d-53c1-4c79-b81b-e804b01be653@googlegroups.com>
 <144b0566-6738-bd37-cd1c-1585e70761d2@gmail.com>
 <mailman.151.1479221023.30068.python-list@python.org>
 <80870bc5-5975-43e0-bf23-dccb1f628d6f@googlegroups.com>
 <5d48ae5f-17f1-fca7-1239-d22c2950363c@gmail.com>
 <mailman.154.1479266093.30068.python-list@python.org>
 <c4c12dd3-7d2d-4a87-bdbc-0e752dbf5318@googlegroups.com>
 <ababec33-ad77-60f4-367f-aa89aad22230@gmail.com>
 <mailman.7.1479397099.20410.python-list@python.org>
 <023fdec4-3630-4030-a58b-24028dde4b41@googlegroups.com>
 <5ee9f8ce-e3a6-8163-02da-f5d928d219e1@gmail.com>
 <mailman.24.1479438177.20410.python-list@python.org>
 <90551ab1-c0b9-48db-85c3-c9487926b3f7@googlegroups.com>
 <e558b448-702c-c8c0-296a-ad60d6cca18d@gmail.com>
 <mailman.27.1479442451.20410.python-list@python.org>
Message-ID: <583073e2$0$1619$c3e8da3$5496439d@news.astraweb.com>

On Fri, 18 Nov 2016 03:14 pm, Michael Torrie wrote:

> On 11/17/2016 08:41 PM, jfong at ms4.hinet.net wrote:
>> The fact that most novices will stumble on Python variable many times
>> until it becomes his "second nature" proves it's different from the
>> human language:-)
> 
> The fact is that most novices don't stumble when dealing with Python
> variables. The nature of name binding vs memory boxes does trip up
> people occasionally, but most grasp it quickly and move on.
> 
> Since you are unwilling to prove your assertion or even provide a single
> example (you did once speak of couches and chairs but your reference
> made little sense), we can make no further progress.

I think you're being harsh on J Fong. And for what it is worth, I think that
I (slightly) agree with him: in my experience, many people have difficulty
understanding object model at first, especially if they've come from a
background of named memory boxes.

You should spend more time around beginners -- it is quite common for them
to be confused by name binding and all its consequences. I don't know if it
is "most novices", but I'm confident that it is quite common.

Coincidentally, on the Python tutor list, there was an example of this
earlier yesterday:

https://mail.python.org/pipermail/tutor/2016-November/110014.html

This is not just Python, of course, it includes most modern scripting
languages and some non-scripting languages. E.g. Lua, Javascript, Ruby and
Java (objects and boxed values, but not machine values). It's not that
Python is doing something weird compared to other languages. Python's name
binding semantics are very common.

They also have a lot in common with pointer semantics in C-like and
Algol-like languages: you have to reason through a level of indirection to
understand variables. And pointer semantics are notoriously difficult. So I
don't think it should be surprising that some people find it hard to
understand, at least at the beginning.





-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From porton at narod.ru  Sat Nov 19 11:21:57 2016
From: porton at narod.ru (Victor Porton)
Date: Sat, 19 Nov 2016 18:21:57 +0200
Subject: C3 MRO
Message-ID: <o0pu74$jvg$1@gioia.aioe.org>

Do I understand correctly, than C3 applies to particular methods, and thus 
it does not fail, if it works for every defined method, even if it can fail 
after addition of a new method?

Also, at which point it fails: at definition of a class or at calling a 
particular "wrong" method?

-- 
Victor Porton - http://portonvictor.org


From rosuav at gmail.com  Sat Nov 19 11:50:28 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 20 Nov 2016 03:50:28 +1100
Subject: C3 MRO
In-Reply-To: <o0pu74$jvg$1@gioia.aioe.org>
References: <o0pu74$jvg$1@gioia.aioe.org>
Message-ID: <CAPTjJmpkjJTf+jNQ6ix=cxidYJLSc-+xEdbSYiNmykCND8VB+Q@mail.gmail.com>

On Sun, Nov 20, 2016 at 3:21 AM, Victor Porton <porton at narod.ru> wrote:
> Do I understand correctly, than C3 applies to particular methods, and thus
> it does not fail, if it works for every defined method, even if it can fail
> after addition of a new method?
>
> Also, at which point it fails: at definition of a class or at calling a
> particular "wrong" method?

It either succeeds or fails for the entire class. If it fails, you get
an exception at class definition time:

class A: pass
class B: pass
class C(A, B): pass
class D(B, A): pass
class E(C, D): pass

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Cannot create a consistent method resolution
order (MRO) for bases A, B

ChrisA


From __peter__ at web.de  Sat Nov 19 12:25:57 2016
From: __peter__ at web.de (Peter Otten)
Date: Sat, 19 Nov 2016 18:25:57 +0100
Subject: Two variants of class hierachy
References: <o0ppdo$bpu$1@gioia.aioe.org>
Message-ID: <o0q1v5$eor$1@blaine.gmane.org>

Victor Porton wrote:

> I am developing software which shows hierarchical information (tree),
> including issues and comments from BitBucket (comments are sub-nodes of
> issues, thus it forms a tree).
> 
> There are two kinds of objects in the hierarchy: a. with a (possibly long)
> paginated list of childs; b. with a short list of strings, each string
> being associated with a child object.
> 
> I have two variants of class inheritance in mind. Please help to decide
> which is better.
> 
> The first one declares only one base class, but some its method remain
> unimplemented (raise NotImplementedError) even in derived classes.

Two observations:

In Python you can also use "duck-typing" -- if you don't want to share code 
between the classes there is no need for an inhertitance tree at all.

Pagination is a matter of display and will be handled differently in a PDF 
document or web page, say. I would not make it part of the data structure.




From porton at narod.ru  Sat Nov 19 13:00:41 2016
From: porton at narod.ru (Victor Porton)
Date: Sat, 19 Nov 2016 20:00:41 +0200
Subject: Two variants of class hierachy
References: <o0ppdo$bpu$1@gioia.aioe.org> <o0q1v5$eor$1@blaine.gmane.org>
 <mailman.51.1479576377.20410.python-list@python.org>
Message-ID: <o0q408$utq$1@gioia.aioe.org>

Peter Otten wrote:

> Victor Porton wrote:
> 
>> I am developing software which shows hierarchical information (tree),
>> including issues and comments from BitBucket (comments are sub-nodes of
>> issues, thus it forms a tree).
>> 
>> There are two kinds of objects in the hierarchy: a. with a (possibly
>> long) paginated list of childs; b. with a short list of strings, each
>> string being associated with a child object.
>> 
>> I have two variants of class inheritance in mind. Please help to decide
>> which is better.
>> 
>> The first one declares only one base class, but some its method remain
>> unimplemented (raise NotImplementedError) even in derived classes.
> 
> Two observations:
> 
> In Python you can also use "duck-typing" -- if you don't want to share
> code between the classes there is no need for an inhertitance tree at all.

I know, but explicit inheritance serves as a kind of documentation for 
readers of my code.

> Pagination is a matter of display and will be handled differently in a PDF
> document or web page, say. I would not make it part of the data structure.

Not in my case, because the data I receive is already paginated. I am not 
going to "re-paginate" it in another way.

-- 
Victor Porton - http://portonvictor.org


From porton at narod.ru  Sat Nov 19 13:16:57 2016
From: porton at narod.ru (Victor Porton)
Date: Sat, 19 Nov 2016 20:16:57 +0200
Subject: Extra base class in hierarchy
Message-ID: <o0q4uo$10u8$1@gioia.aioe.org>

Consider

class FinalTreeNode(object):
    def childs(self):
        return []

class UsualTreeNode(FinalTreeNode)
    def childs(self):
        return ...

In this structure UsualTreeNode derives from FinalTreeNode.

Is it better to introduce an extra base class?

class BaseTreeNode(object):
    def childs(self):
        return []

# The same functionality as BaseTreeNode, but logically distinct
class FinalTreeNode(BaseTreeNode):
    pass

# Not derived from FinalTreeNode, because it is not logically final
class UsualTreeNode(BaseTreeNode)
    def childs(self):
        return ...

-- 
Victor Porton - http://portonvictor.org


From ian.g.kelly at gmail.com  Sat Nov 19 14:34:57 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Sat, 19 Nov 2016 12:34:57 -0700
Subject: Extra base class in hierarchy
In-Reply-To: <o0q4uo$10u8$1@gioia.aioe.org>
References: <o0q4uo$10u8$1@gioia.aioe.org>
Message-ID: <CALwzidmsM8=ApKBJAjcR+Jj5DAbDHLB-=nQbxTqhPsAQmZrzJQ@mail.gmail.com>

On Nov 19, 2016 11:22 AM, "Victor Porton" <porton at narod.ru> wrote:

Consider

class FinalTreeNode(object):
    def childs(self):
        return []

class UsualTreeNode(FinalTreeNode)
    def childs(self):
        return ...

In this structure UsualTreeNode derives from FinalTreeNode.


This looks odd because "final" in OOP usually denotes something that will
not be further inherited.

Is it better to introduce an extra base class?


Without knowing what UsualTreeNode and FinalTreeNode are it's hard to say.


class BaseTreeNode(object):
    def childs(self):
        return []

# The same functionality as BaseTreeNode, but logically distinct
class FinalTreeNode(BaseTreeNode):
    pass


That's a bit of a code smell. Why do you need a subclass if the
implementation is the same? Probably you should implement the common parts
in the base class and differentiate both of the subclasses.

Also a pet peeve of mine is the word "base" in class names. Just call it
TreeNode.


From saxri89 at gmail.com  Sat Nov 19 15:44:49 2016
From: saxri89 at gmail.com (Xristos Xristoou)
Date: Sat, 19 Nov 2016 12:44:49 -0800 (PST)
Subject: Couldn't load SIP module.
Message-ID: <1c1eb1cf-a24a-47ea-8af9-6a7538c7cf1b@googlegroups.com>

hello
i using python 2.7.12 on ubuntu 16.04 and i have that error with SIP :

Couldn't load SIP module.


Traceback (most recent call last):
  File "", line 1, in 
ImportError: /usr/local/lib/python2.7/site-packages/sip.so: undefined symbol: PyUnicodeUCS2_AsUTF8String

('Qt version:', '4.8.7')
('SIP version:', '4.17')
('PyQt version:', '4.11.4')

any idea how to fix that ?


From greg.ewing at canterbury.ac.nz  Sat Nov 19 16:25:36 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Sun, 20 Nov 2016 10:25:36 +1300
Subject: Extra base class in hierarchy
In-Reply-To: <mailman.52.1479584101.20410.python-list@python.org>
References: <o0q4uo$10u8$1@gioia.aioe.org>
 <CALwzidmsM8=ApKBJAjcR+Jj5DAbDHLB-=nQbxTqhPsAQmZrzJQ@mail.gmail.com>
 <mailman.52.1479584101.20410.python-list@python.org>
Message-ID: <e9buaiFokd3U1@mid.individual.net>

Ian Kelly wrote:
> Is it better to introduce an extra base class?

That's one possibility. An advantage would be that it
would be easier to add methods in the future that apply
to UsualTreeNodes but not FinalTreeNodes.

Another possibility would be to just rename the classes.
Instead of FinalTreeNode and UsualTreeNode, call them
something like TreeNode and NonFinalTreeNode, with
NonFinalTreeNode inheriting from TreeNode.

-- 
Greg


From jfong at ms4.hinet.net  Sat Nov 19 21:31:40 2016
From: jfong at ms4.hinet.net (jfong at ms4.hinet.net)
Date: Sat, 19 Nov 2016 18:31:40 -0800 (PST)
Subject: How to append a modified list into a list?
In-Reply-To: <mailman.49.1479548417.20410.python-list@python.org>
References: <cbba816d-8bfd-4578-a58d-d12d33a9ff69@googlegroups.com>
 <o0p6lj$1eh$1@blaine.gmane.org>
 <mailman.49.1479548417.20410.python-list@python.org>
Message-ID: <a82981df-fd16-4458-bae6-d722baf18cb3@googlegroups.com>

Peter Otten at 2016/11/19 5:40:34PM wrote:
> And now for something completely different ;)
> 
> What if you only record the changes to the list? For a long list that would 
> save space at the expense of calculation time. For example:

Excellent! Although not 100% fit into my application, I must study how this Class works. Thank you.

--Jach


From torriem at gmail.com  Sat Nov 19 21:54:54 2016
From: torriem at gmail.com (Michael Torrie)
Date: Sat, 19 Nov 2016 19:54:54 -0700
Subject: help on "from deen import *" vs. "import deen"
In-Reply-To: <583073e2$0$1619$c3e8da3$5496439d@news.astraweb.com>
References: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>
 <mailman.147.1479180690.30068.python-list@python.org>
 <1a6f991d-53c1-4c79-b81b-e804b01be653@googlegroups.com>
 <144b0566-6738-bd37-cd1c-1585e70761d2@gmail.com>
 <mailman.151.1479221023.30068.python-list@python.org>
 <80870bc5-5975-43e0-bf23-dccb1f628d6f@googlegroups.com>
 <5d48ae5f-17f1-fca7-1239-d22c2950363c@gmail.com>
 <mailman.154.1479266093.30068.python-list@python.org>
 <c4c12dd3-7d2d-4a87-bdbc-0e752dbf5318@googlegroups.com>
 <ababec33-ad77-60f4-367f-aa89aad22230@gmail.com>
 <mailman.7.1479397099.20410.python-list@python.org>
 <023fdec4-3630-4030-a58b-24028dde4b41@googlegroups.com>
 <5ee9f8ce-e3a6-8163-02da-f5d928d219e1@gmail.com>
 <mailman.24.1479438177.20410.python-list@python.org>
 <90551ab1-c0b9-48db-85c3-c9487926b3f7@googlegroups.com>
 <e558b448-702c-c8c0-296a-ad60d6cca18d@gmail.com>
 <mailman.27.1479442451.20410.python-list@python.org>
 <583073e2$0$1619$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <eee3a048-1d38-8701-596b-609f4535495f@gmail.com>

On 11/19/2016 08:46 AM, Steve D'Aprano wrote:
> I think you're being harsh on J Fong. And for what it is worth, I think that
> I (slightly) agree with him: in my experience, many people have difficulty
> understanding object model at first, especially if they've come from a
> background of named memory boxes.
> 
> You should spend more time around beginners -- it is quite common for them
> to be confused by name binding and all its consequences. I don't know if it
> is "most novices", but I'm confident that it is quite common.

Yes, I can see that you're correct. I apologize to Mr Fong for my
impatience.  Python is pretty cool, and I hope it starts to click for you.



From torriem at gmail.com  Sat Nov 19 23:24:59 2016
From: torriem at gmail.com (Michael Torrie)
Date: Sat, 19 Nov 2016 21:24:59 -0700
Subject: Couldn't load SIP module.
In-Reply-To: <1c1eb1cf-a24a-47ea-8af9-6a7538c7cf1b@googlegroups.com>
References: <1c1eb1cf-a24a-47ea-8af9-6a7538c7cf1b@googlegroups.com>
Message-ID: <46a59bde-a60f-86e9-a14e-f958002ec122@gmail.com>

On 11/19/2016 01:44 PM, Xristos Xristoou wrote:
> hello
> i using python 2.7.12 on ubuntu 16.04 and i have that error with SIP :
> 
> Couldn't load SIP module.
> 
> 
> Traceback (most recent call last):
>   File "", line 1, in 
> ImportError: /usr/local/lib/python2.7/site-packages/sip.so: undefined symbol: PyUnicodeUCS2_AsUTF8String
> 
> ('Qt version:', '4.8.7')
> ('SIP version:', '4.17')
> ('PyQt version:', '4.11.4')
> 
> any idea how to fix that ?

Looks like you're playing with a custom-compiled version of Python, as
/usr/local/python2.7 is not a normal Ubuntu path for Python to be
installed as far as I know.  Feel free to correct me.  I suspect there's
some impedance mismatch between your custom version of Python and the
system-supplied libraries that sip.so may be depending on.

I'm pretty sure Ubuntu 16.04 ships with a pretty current version of
Python 2.7 and also Python 3.  It will probably be way easier to work
with the Ubuntu-provided Python versions than to compile you're own,
unless you have a good reason to do so.  Try deleting your custom
installation of Python and use the Ubuntu software manager or apt-get to
install PyQt and the sip dependencies.



From thorsten at thorstenkampe.de  Sun Nov 20 04:27:11 2016
From: thorsten at thorstenkampe.de (Thorsten Kampe)
Date: Sun, 20 Nov 2016 10:27:11 +0100
Subject: Generic dictionary
Message-ID: <o0rqae$t10$1@blaine.gmane.org>

[Crossposted to tutor and general mailing list]

Hi,

I'd like to extend the dictionary class by creating a class that acts 
like a dictionary if the class is instantiated with a dictionary and 
acts like a "dictitem" ([(key1, value1), (key2, value2), ...]) if 
instantiated with a list (that is dictitem).

The code (see extract at bottom) works well but it contains a lot of 
"if this is a dictionary then do as a dictionary already does" 
boilerplate code". How can I "inherit"(?)/"subclass"(?)/derive from 
dict so I don't have to write the code for the dictionary case?

Thorsten

```
class GenericDict:
    """
    a GenericDict is a dictionary or a list of tuples (when the keys
    are not hashable)
    """
    def __init__(inst, generic_dict):
        inst._generic = generic_dict

    def __getitem__(inst, key):
        if isinstance(inst._generic, dict):
            return inst._generic[key]
        else:
            return inst.values()[inst.keys().index(key)]

    def values(inst):
        if isinstance(inst._generic, dict):
            return inst._generic.values()
        else:
            try:
                return list(zip(*inst._generic))[1]
            except IndexError:  # empty GenericDict
                return ()

    def keys(inst):
        if isinstance(inst._generic, dict):
            return inst._generic.keys()
        else:
            try:
                return list(zip(*inst._generic))[0]
            except IndexError:  # empty GenericDict
                return ()
```



From __peter__ at web.de  Sun Nov 20 04:43:01 2016
From: __peter__ at web.de (Peter Otten)
Date: Sun, 20 Nov 2016 10:43:01 +0100
Subject: Generic dictionary
References: <o0rqae$t10$1@blaine.gmane.org>
Message-ID: <o0rr71$4qr$3@blaine.gmane.org>

Thorsten Kampe wrote:

> [Crossposted to tutor and general mailing list]
> 
> Hi,
> 
> I'd like to extend the dictionary class by creating a class that acts
> like a dictionary if the class is instantiated with a dictionary and
> acts like a "dictitem" ([(key1, value1), (key2, value2), ...]) if
> instantiated with a list (that is dictitem).
> 
> The code (see extract at bottom) works well but it contains a lot of
> "if this is a dictionary then do as a dictionary already does"
> boilerplate code". How can I "inherit"(?)/"subclass"(?)/derive from
> dict so I don't have to write the code for the dictionary case?

Hm.

def GenericDict(dict_or_items):
    if isinstance(dict_or_items, dict):
        return dict(dict_or_items)
    else:
        return SimpleGenericDictWithOnlyTheFalseBranchesImplemented(
            dict_or_items
        )

> 
> Thorsten
> 
> ```
> class GenericDict:
>     """
>     a GenericDict is a dictionary or a list of tuples (when the keys
>     are not hashable)
>     """
>     def __init__(inst, generic_dict):
>         inst._generic = generic_dict
> 
>     def __getitem__(inst, key):
>         if isinstance(inst._generic, dict):
>             return inst._generic[key]
>         else:
>             return inst.values()[inst.keys().index(key)]

It's not obvious why you'd ever want that. What's the use case?
If you have unhashable keys consider bisect...
 
>     def values(inst):
>         if isinstance(inst._generic, dict):
>             return inst._generic.values()
>         else:
>             try:
>                 return list(zip(*inst._generic))[1]
>             except IndexError:  # empty GenericDict
>                 return ()
> 
>     def keys(inst):
>         if isinstance(inst._generic, dict):
>             return inst._generic.keys()
>         else:
>             try:
>                 return list(zip(*inst._generic))[0]
>             except IndexError:  # empty GenericDict
>                 return ()
> ```



From saxri89 at gmail.com  Sun Nov 20 04:52:04 2016
From: saxri89 at gmail.com (Xristos Xristoou)
Date: Sun, 20 Nov 2016 01:52:04 -0800 (PST)
Subject: Couldn't load SIP module.
In-Reply-To: <1c1eb1cf-a24a-47ea-8af9-6a7538c7cf1b@googlegroups.com>
References: <1c1eb1cf-a24a-47ea-8af9-6a7538c7cf1b@googlegroups.com>
Message-ID: <72cdd9ee-8702-445c-954b-a9abbba986a7@googlegroups.com>

?? ???????, 19 ????????? 2016 - 10:45:16 ?.?. UTC+2, ? ??????? Xristos Xristoou ??????:
> hello
> i using python 2.7.12 on ubuntu 16.04 and i have that error with SIP :
> 
> Couldn't load SIP module.
> 
> 
> Traceback (most recent call last):
>   File "", line 1, in 
> ImportError: /usr/local/lib/python2.7/site-packages/sip.so: undefined symbol: PyUnicodeUCS2_AsUTF8String
> 
> ('Qt version:', '4.8.7')
> ('SIP version:', '4.17')
> ('PyQt version:', '4.11.4')
> 
> any idea how to fix that ?

how to delete complete python with all packages on ubuntu ?


From steve+python at pearwood.info  Sun Nov 20 05:10:08 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sun, 20 Nov 2016 21:10:08 +1100
Subject: Generic dictionary
References: <o0rqae$t10$1@blaine.gmane.org>
 <mailman.0.1479634074.3146.python-list@python.org>
Message-ID: <58317681$0$1599$c3e8da3$5496439d@news.astraweb.com>

On Sun, 20 Nov 2016 08:27 pm, Thorsten Kampe wrote:

> I'd like to extend the dictionary class by creating a class that acts
> like a dictionary if the class is instantiated with a dictionary and
> acts like a "dictitem" ([(key1, value1), (key2, value2), ...]) if
> instantiated with a list (that is dictitem).

I'm not sure exactly what you are trying to accomplish here. You want a
single class that sometimes behaves like a dict, and sometimes behaves like
a list? Not just any list, but specifically a list of tuples of two items.

Frankly, that sounds... weird. Generally speaking, classes don't behave
differently according to how they are created. Consider:

float(123)  # int argument

float("123")  # str argument


The object you get back is the same, regardless of whether the class was
instantiated from an int or a str. That's a general principle of Object
Oriented Programming: an object's behaviour should depend on *what it is*,
not where it came from.


> The code (see extract at bottom) works well but it contains a lot of
> "if this is a dictionary then do as a dictionary already does"
> boilerplate code". How can I "inherit"(?)/"subclass"(?)/derive from
> dict so I don't have to write the code for the dictionary case?

class MyDict(dict):
    pass


inherits from dict. But that won't do what you want, since a dict doesn't
behave anything like a list of tuples. You'll have to over-ride
*everything*, filling your code with horrible stuff like this:


class MyDict(dict):
    def __init__(self, *args, **kwargs):
        super(MyDict, self).__init__(*args, **kwargs)
        # Decide whether to act like a list or a dict.
        if args and isinstance(args[0], dict):
            self._act_like_dict = True
        else:
            self._act_like_dict = False

    def __getitem__(self, key_or_index):
        if self._act_like_dict:
            # treat like a key
            super(MyDict, self).__getitem__(key_or_index)
         else:
            # treat like an index
            tmp = list(self.values())
            return tmp[key_or_index]

    def update(self, *args, **kwargs):
        if self._act_like_dict:
            super(MyDict, self).update(*args, **kwargs)
         else:
            raise TypeError('list doesn't support update')

    def sort(self, *args, **kwargs):
        if self._act_like_dict:
            raise TypeError('dict doesn't support sort')
         else:
            # somehow re-implement list sorting for something 
            # which is like a dict
            ...



Etc. It would be a monstrous pain in the posterior to write, and confusing
to use. Are you sure you want this? The whole concept of a class that
behaves differently depending on how you create it feels to me like that
movie "The Fly".

I think that a better strategy would be to reconsider. Why not just write
your code to accept *either* a dict, or a list of tuples? Then you don't
have to invent some monstrous hybrid class that is like the product of an
awful teleportation accident.






-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From b1540457 at tyldd.com  Sun Nov 20 05:46:25 2016
From: b1540457 at tyldd.com (Anny Mous)
Date: Sun, 20 Nov 2016 21:46:25 +1100
Subject: Generic dictionary
References: <o0rqae$t10$1@blaine.gmane.org> <o0rr71$4qr$3@blaine.gmane.org>
Message-ID: <o0rutu$sr4$1@blaine.gmane.org>

On Sun, 20 Nov 2016 08:43 pm, Peter Otten wrote:

> Thorsten Kampe wrote:
> 
>> [Crossposted to tutor and general mailing list]
>> 
>> Hi,
>> 
>> I'd like to extend the dictionary class by creating a class that acts
>> like a dictionary if the class is instantiated with a dictionary and
>> acts like a "dictitem" ([(key1, value1), (key2, value2), ...]) if
>> instantiated with a list (that is dictitem).
[...]
> def GenericDict(dict_or_items):
>     if isinstance(dict_or_items, dict):
>         return dict(dict_or_items)
>     else:
>         return SimpleGenericDictWithOnlyTheFalseBranchesImplemented(
>             dict_or_items
>         )


Personally, I'd go even simpler:

dict(dict_of_items)

will return a dict regardless of whether you start with another dict or a
list or tuples. And then you just work on it as if it were a dict (which is
exactly what it actually is). And if you want to turn it back into a list
of (key, value) pairs, then you call:

list(d.items())  # Python 3

d.items()  # Python 2



-- 
Steve




From steve+python at pearwood.info  Sun Nov 20 06:40:19 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sun, 20 Nov 2016 22:40:19 +1100
Subject: Generic dictionary
References: <o0rqae$t10$1@blaine.gmane.org>
 <mailman.0.1479634074.3146.python-list@python.org>
Message-ID: <58318ba5$0$1588$c3e8da3$5496439d@news.astraweb.com>

Further thoughts come to mind, after looking more closely at your code.

On Sun, 20 Nov 2016 08:27 pm, Thorsten Kampe wrote:

> class GenericDict:
>     """
>     a GenericDict is a dictionary or a list of tuples (when the keys
>     are not hashable)
>     """
>     def __init__(inst, generic_dict):
>         inst._generic = generic_dict
> 
>     def __getitem__(inst, key):
>         if isinstance(inst._generic, dict):
>             return inst._generic[key]
>         else:
>             return inst.values()[inst.keys().index(key)]

Your code seems to be different from your description. Rather than
*behaving* like a list of (key, item) pairs, you seem to actually have
something which always behaves like a dict, but has a different internal
storage. That's less disturbing than what I thought you wanted at first.

There's probably a "Design Pattern" name for this sort of thing: a
consistent front end, with a different back end for storage. But I guess it
is overkill for here, and besides, using a list of (key, item) is not going
to be an efficient back end storage. Oh, it will be fine if you have ten or
a hundred items, but if you have a million, it will be painful.

Better to just use a dict, always, and simply convert any list of tuples to
a dict:

py> dict([('a', 1), ('b', 2), ('c', 3)])
{'c': 3, 'a': 1, 'b': 2}


>     def values(inst):
>         if isinstance(inst._generic, dict):
>             return inst._generic.values()
>         else:
>             try:
>                 return list(zip(*inst._generic))[1]
>             except IndexError:  # empty GenericDict
>                 return ()

That's... weird. Sometimes your instance will return a list, and sometimes a
tuple. So if you write code expecting a list, it will suddenly and
unexpectedly break when the instance is empty.

A better way of writing this is:

    def values(inst):
        if isinstance(inst._generic, dict):
            return inst._generic.values()
        else:
            return [t[1] for t in inst._generic]

(But even better is not to write it at all, and just use the dict, like I
suggested above :-)





-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From countryone77 at gmail.com  Sun Nov 20 06:47:24 2016
From: countryone77 at gmail.com (Bev in TX)
Date: Sun, 20 Nov 2016 05:47:24 -0600
Subject: help on "from deen import *" vs. "import deen"
In-Reply-To: <mailman.48.1479538706.20410.python-list@python.org>
References: <5d48ae5f-17f1-fca7-1239-d22c2950363c@gmail.com>
 <mailman.154.1479266093.30068.python-list@python.org>
 <c4c12dd3-7d2d-4a87-bdbc-0e752dbf5318@googlegroups.com>
 <ababec33-ad77-60f4-367f-aa89aad22230@gmail.com>
 <mailman.7.1479397099.20410.python-list@python.org>
 <023fdec4-3630-4030-a58b-24028dde4b41@googlegroups.com>
 <5ee9f8ce-e3a6-8163-02da-f5d928d219e1@gmail.com>
 <mailman.24.1479438177.20410.python-list@python.org>
 <90551ab1-c0b9-48db-85c3-c9487926b3f7@googlegroups.com>
 <kaiu2cdjgd8bu94tvhvve532ug3v8r3uf6@4ax.com>
 <mailman.40.1479495082.20410.python-list@python.org>
 <582fd63b$0$1598$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmroWO8iKnFRrGA_9yjKMfJdiuPpDhM0a3jxZ=QZztJHgA@mail.gmail.com>
 <mailman.48.1479538706.20410.python-list@python.org>
Message-ID: <3f64eee1-cfd8-a4a4-ff7f-f8f68505dea0@gmail.com>

 From the Python 3.5.2 docs:

6.15. Evaluation order
Python evaluates expressions from left to right. Notice that while 
evaluating an assignment, the right-hand side is evaluated before the 
left-hand side.

Thus, spam = eggs = cheese = obj is equivalent to:

spam = (eggs = (cheese = obj))

which is also equivalent to:

cheese = obj
eggs = cheese
spam = eggs

These all are now references to obj.

In both of your examples, everything also ends up as references to obj. 
The only difference is the order of evaluation.

Bev in TX

On 11/19/16 12:58 AM, Chris Angelico wrote:
> On Sat, Nov 19, 2016 at 3:34 PM, Steve D'Aprano
> <steve+python at pearwood.info> wrote:
>> What happens if you do this?
>>
>> spam = eggs = cheese = obj
>>
>> Is that different from:
>>
>> spam = obj
>> eggs = obj
>> cheese = obj
>>
>>
>> or from this?
>>
>> spam = obj
>> eggs = spam
>> cheese = eggs
>> ...
>> These aren't silly questions.
>
> Indeed, they are not silly. It surprised me (as a C programmer) that
> the assignments happen left-to-right, rather than right-to-left (in C,
> cheese would be set first, then eggs, then spam). These are questions
> filled with subtleties.
>
> ChrisA
>


From thorsten at thorstenkampe.de  Sun Nov 20 07:12:51 2016
From: thorsten at thorstenkampe.de (Thorsten Kampe)
Date: Sun, 20 Nov 2016 13:12:51 +0100
Subject: Generic dictionary
References: <o0rqae$t10$1@blaine.gmane.org> <o0rr71$4qr$3@blaine.gmane.org>
Message-ID: <o0s412$hnq$1@blaine.gmane.org>

* Peter Otten (Sun, 20 Nov 2016 10:43:01 +0100)
> 
> Thorsten Kampe wrote:
> > 
> > I'd like to extend the dictionary class by creating a class that acts
> > like a dictionary if the class is instantiated with a dictionary and
> > acts like a "dictitem" ([(key1, value1), (key2, value2), ...]) if
> > instantiated with a list (that is dictitem).
> > 
> > The code (see extract at bottom) works well but it contains a lot of
> > "if this is a dictionary then do as a dictionary already does"
> > boilerplate code". How can I "inherit"(?)/"subclass"(?)/derive from
> > dict so I don't have to write the code for the dictionary case?
> 
> Hm.
> 
> def GenericDict(dict_or_items):
>     if isinstance(dict_or_items, dict):
>         return dict(dict_or_items)
>     else:
>         return SimpleGenericDictWithOnlyTheFalseBranchesImplemented(
>             dict_or_items
>         )

That would be a kind of factory function for the class?
 
> > ```
> > class GenericDict:
> >     """
> >     a GenericDict is a dictionary or a list of tuples (when the keys
> >     are not hashable)
> >     """
> >     def __init__(inst, generic_dict):
> >         inst._generic = generic_dict
> > 
> >     def __getitem__(inst, key):
> >         if isinstance(inst._generic, dict):
> >             return inst._generic[key]
> >         else:
> >             return inst.values()[inst.keys().index(key)]
> 
> It's not obvious why you'd ever want that. What's the use case?

The use case is an Equivalence class which creates a {invariant(x): 
[x, y, z] ...} dictionary. Since the Equivalence class should accept 
all kinds of key functions, I have to make sure that lists (which are 
not hashable), etc. can be handled as keys.

Treating a list of tuples as a substitute for a dictionary is a well 
established idiom (think list(dict.items()) and dict()).

The GenericDict class allows the Equivalence class to be agnostic 
regarding the underlying data structure. It tries to create a 
dictionary and if that fails uses a dictitem. All method calls are 
the same because that's handled by GenericDict.

Thorsten



From thorsten at thorstenkampe.de  Sun Nov 20 07:19:03 2016
From: thorsten at thorstenkampe.de (Thorsten Kampe)
Date: Sun, 20 Nov 2016 13:19:03 +0100
Subject: Generic dictionary
References: <o0rqae$t10$1@blaine.gmane.org> <o0rr71$4qr$3@blaine.gmane.org>
 <o0rutu$sr4$1@blaine.gmane.org>
Message-ID: <o0s4cm$hnq$2@blaine.gmane.org>

* Anny Mous (Sun, 20 Nov 2016 21:46:25 +1100)
> 
> On Sun, 20 Nov 2016 08:43 pm, Peter Otten wrote:
> 
> > Thorsten Kampe wrote:
> > 
> >> [Crossposted to tutor and general mailing list]
> >> 
> >> Hi,
> >> 
> >> I'd like to extend the dictionary class by creating a class that acts
> >> like a dictionary if the class is instantiated with a dictionary and
> >> acts like a "dictitem" ([(key1, value1), (key2, value2), ...]) if
> >> instantiated with a list (that is dictitem).
> [...]
> > def GenericDict(dict_or_items):
> >     if isinstance(dict_or_items, dict):
> >         return dict(dict_or_items)
> >     else:
> >         return SimpleGenericDictWithOnlyTheFalseBranchesImplemented(
> >             dict_or_items
> >         )
> 
> 
> Personally, I'd go even simpler:
> 
> dict(dict_of_items)
> 
> will return a dict regardless of whether you start with another dict or a
> list or tuples.

The whole point of my posting was non hashable keys (like lists):

```
>>> dictitem
[([1], '11'), ([2], '22'), ([4], '33'), ([3], '44')]
>>> dict(dictitem)
---------------------------------------------------------------------
TypeError                           Traceback (most recent call last)
<ipython-input-12-0f2b626ac851> in <module>()
----> 1 dict(dictitem)

TypeError: unhashable type: 'list'
```

Thorsten



From deavmi at disroot.org  Sun Nov 20 07:22:54 2016
From: deavmi at disroot.org (Tristan B. Kildaire)
Date: Sun, 20 Nov 2016 12:22:54 -0000 (UTC)
Subject: help on "from deen import *" vs. "import deen"
References: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>
 <ca14fafb-2226-493e-ada5-e850a3c74f48@googlegroups.com>
 <e0478664-9610-4d83-8f58-583682d11fcd@googlegroups.com>
 <bb816fbf-1df7-522d-bab7-9f7a65deaef3@mrabarnett.plus.com>
 <mailman.147.1479180690.30068.python-list@python.org>
 <1a6f991d-53c1-4c79-b81b-e804b01be653@googlegroups.com>
 <144b0566-6738-bd37-cd1c-1585e70761d2@gmail.com>
 <9b6ad48d-ee65-bca6-6fa7-4176ddaae83e@lucidity.plus.com>
 <mailman.153.1479248175.30068.python-list@python.org>
Message-ID: <o0s4it$c8a$2@dont-email.me>

On Tue, 15 Nov 2016 22:16:07 +0000, Erik wrote:

> On 15/11/16 14:43, Michael Torrie wrote:
>> As you've been told several times, if you "import deen" then you can
>> place a new object into the deen namespace using something like:
>>
>> deen.foo=bar
>>
>> Importing everything from an imported module into the current module's
>> namespace is not the best idea
> 
> But "from foo import *" is not importing "everything". It's the opposite
> - it's importing everything _that the module wants to explicitly expose_
> (i.e., a subset of everything ;)).
> 
> It *used* to be everything in the module's namespace, but then the
> possibly misnamed "__all__" magic variable made it a subset of whatever
> the module's author wanted to expose.
> 
> However, "import foo" _does_ import "everything", and also gives the
> importer the power to re-bind names within that module's namespace too
> (which is the crux of the original question). This is not usually a good
> thing unless the importing module is incestuous with the imported
> module.
> 
> So this brings up another point - not sure if it has been made before:
> should a module have a way of *not* allowing an importer to re-bind its
> local bindings?
> 
> For example, something like a "__bindable__" variable such that all
> names not included may not be bound by any other module? And/or
> something like an "__export__" variable that says what gets exposed to a
> simple "import foo" (sometimes one might want to "import foo" and
> sometimes "from foo import *" for namespace reasons, but why should
> those two statements import different things)?
> 
> E.

>From deen import * imports all the things in deen but accessable with no 
`deen.`

import deen imports all of deen but accessible via `deen.thing`


From deavmi at disroot.org  Sun Nov 20 07:24:01 2016
From: deavmi at disroot.org (Tristan B. Kildaire)
Date: Sun, 20 Nov 2016 12:24:01 -0000 (UTC)
Subject: Guido? Where are you?
Message-ID: <o0s4l0$c8a$3@dont-email.me>

Is Guido active on this newsgroup. Sorry for the off-topic ness.


From thorsten at thorstenkampe.de  Sun Nov 20 07:26:24 2016
From: thorsten at thorstenkampe.de (Thorsten Kampe)
Date: Sun, 20 Nov 2016 13:26:24 +0100
Subject: Generic dictionary
References: <o0rqae$t10$1@blaine.gmane.org>
 <mailman.0.1479634074.3146.python-list@python.org>
 <58317681$0$1599$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o0s4qf$hnq$3@blaine.gmane.org>

* Steve D'Aprano (Sun, 20 Nov 2016 21:10:08 +1100)
> 
> On Sun, 20 Nov 2016 08:27 pm, Thorsten Kampe wrote:
> 
> > I'd like to extend the dictionary class by creating a class that acts
> > like a dictionary if the class is instantiated with a dictionary and
> > acts like a "dictitem" ([(key1, value1), (key2, value2), ...]) if
> > instantiated with a list (that is dictitem).
> 
> I'm not sure exactly what you are trying to accomplish here. You want a
> single class that sometimes behaves like a dict, and sometimes behaves like
> a list? Not just any list, but specifically a list of tuples of two items.

Treating a list of tuples as a substitute for a dictionary is a well 
established idiom (think list(dict.items()) and dict()).
 
> Frankly, that sounds... weird. Generally speaking, classes don't behave
> differently according to how they are created.

I think, you misinterpreted my question. It's not about behaving 
differently but the opposite: a streamlined interface for the caller 
so the caller does not have to worry whether it uses a dict or a 
dictitem.

See my response to Peter:
The use case is an Equivalence class which creates a {invariant(x): 
[x, y, z] ...} dictionary. Since the Equivalence class should accept 
all kinds of key functions, I have to make sure that lists (which are 
not hashable), etc. can be handled as keys.

The GenericDict class allows the Equivalence class to be agnostic 
regarding the underlying data structure. It tries to create a 
dictionary and if that fails uses a dictitem. All method calls are 
the same because that's handled by GenericDict.

Thorsten



From rosuav at gmail.com  Sun Nov 20 07:35:52 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 20 Nov 2016 23:35:52 +1100
Subject: Generic dictionary
In-Reply-To: <o0s4cm$hnq$2@blaine.gmane.org>
References: <o0rqae$t10$1@blaine.gmane.org> <o0rr71$4qr$3@blaine.gmane.org>
 <o0rutu$sr4$1@blaine.gmane.org> <o0s4cm$hnq$2@blaine.gmane.org>
Message-ID: <CAPTjJmqff2oH9-jo7Gs5Qguu8Lg0AZ_u5V6ZV-o+oShbQj3wPg@mail.gmail.com>

On Sun, Nov 20, 2016 at 11:19 PM, Thorsten Kampe
<thorsten at thorstenkampe.de> wrote:
> The whole point of my posting was non hashable keys (like lists):
>
> ```
>>>> dictitem
> [([1], '11'), ([2], '22'), ([4], '33'), ([3], '44')]
>>>> dict(dictitem)
> ---------------------------------------------------------------------
> TypeError                           Traceback (most recent call last)
> <ipython-input-12-0f2b626ac851> in <module>()
> ----> 1 dict(dictitem)
>
> TypeError: unhashable type: 'list'
> ```

I see. So you want to be able to have something that looks and feels
like a dictionary, but uses a different way of looking things up.
Makes reasonable sense, on the surface.

Before you go down that route, I strongly recommend reading up on
exactly *why* a dictionary has the requirement of hashability, and
what the consequences are of trying to look things up using lists as
keys. You can easily experiment with it like this:

class HashableList(list):
    def __hash__(self):
        return hash(tuple(self))

Use those in your dict, rather than vanilla lists. Then you can mess
around with the consequences of mutable dict keys.

Alternatively, switch from using lists to using tuples. They're
hashable and immutable, thus avoiding the problems. If what you're
trying to do is use multi-part dict keys, a tuple is far and away the
best solution.

ChrisA


From rosuav at gmail.com  Sun Nov 20 07:42:50 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 20 Nov 2016 23:42:50 +1100
Subject: help on "from deen import *" vs. "import deen"
In-Reply-To: <3f64eee1-cfd8-a4a4-ff7f-f8f68505dea0@gmail.com>
References: <5d48ae5f-17f1-fca7-1239-d22c2950363c@gmail.com>
 <mailman.154.1479266093.30068.python-list@python.org>
 <c4c12dd3-7d2d-4a87-bdbc-0e752dbf5318@googlegroups.com>
 <ababec33-ad77-60f4-367f-aa89aad22230@gmail.com>
 <mailman.7.1479397099.20410.python-list@python.org>
 <023fdec4-3630-4030-a58b-24028dde4b41@googlegroups.com>
 <5ee9f8ce-e3a6-8163-02da-f5d928d219e1@gmail.com>
 <mailman.24.1479438177.20410.python-list@python.org>
 <90551ab1-c0b9-48db-85c3-c9487926b3f7@googlegroups.com>
 <kaiu2cdjgd8bu94tvhvve532ug3v8r3uf6@4ax.com>
 <mailman.40.1479495082.20410.python-list@python.org>
 <582fd63b$0$1598$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmroWO8iKnFRrGA_9yjKMfJdiuPpDhM0a3jxZ=QZztJHgA@mail.gmail.com>
 <mailman.48.1479538706.20410.python-list@python.org>
 <3f64eee1-cfd8-a4a4-ff7f-f8f68505dea0@gmail.com>
Message-ID: <CAPTjJmpzQgf=bC4c4m9uvocHTS016qoW8sFr9hD4O05r1VbEuw@mail.gmail.com>

On Sun, Nov 20, 2016 at 10:47 PM, Bev in TX <countryone77 at gmail.com> wrote:
> From the Python 3.5.2 docs:
>
> 6.15. Evaluation order
> Python evaluates expressions from left to right. Notice that while
> evaluating an assignment, the right-hand side is evaluated before the
> left-hand side.
>
> Thus, spam = eggs = cheese = obj is equivalent to:
>
> spam = (eggs = (cheese = obj))

Except that that's not how it's parsed. Assignment in Python isn't an
operator. You cannot run the parenthesized version:

>>> spam = (eggs = (cheese = obj))
  File "<stdin>", line 1
    spam = (eggs = (cheese = obj))
                 ^
SyntaxError: invalid syntax

Chained assignment is a special piece of syntax.

https://docs.python.org/3/reference/simple_stmts.html#assignment-statements

You're not evaluating one assignment and then another; it's a single
assignment statement that has a target_list. Scroll down a little in
that page and you'll see an example that specifically points this out.

ChrisA


From ned at nedbatchelder.com  Sun Nov 20 07:43:44 2016
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Sun, 20 Nov 2016 04:43:44 -0800 (PST)
Subject: Guido? Where are you?
In-Reply-To: <o0s4l0$c8a$3@dont-email.me>
References: <o0s4l0$c8a$3@dont-email.me>
Message-ID: <905eaa5c-91a0-4eb5-81df-0ef340e5a7b8@googlegroups.com>

On Sunday, November 20, 2016 at 7:24:43 AM UTC-5, Tristan B. Kildaire wrote:
> Is Guido active on this newsgroup. Sorry for the off-topic ness.

He is definitely not active on this list. The most recent message from
him here (not cross-posted) seems to be in September 2014.

Maybe there's something the rest of us can help you with?

--Ned.


From thorsten at thorstenkampe.de  Sun Nov 20 07:49:38 2016
From: thorsten at thorstenkampe.de (Thorsten Kampe)
Date: Sun, 20 Nov 2016 13:49:38 +0100
Subject: Generic dictionary
References: <o0rqae$t10$1@blaine.gmane.org>
 <mailman.0.1479634074.3146.python-list@python.org>
 <58318ba5$0$1588$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o0s662$hnq$4@blaine.gmane.org>

* Steve D'Aprano (Sun, 20 Nov 2016 22:40:19 +1100)
> 
> Further thoughts come to mind, after looking more closely at your code.
> 
> On Sun, 20 Nov 2016 08:27 pm, Thorsten Kampe wrote:
> 
> >     def values(inst):
> >         if isinstance(inst._generic, dict):
> >             return inst._generic.values()
> >         else:
> >             try:
> >                 return list(zip(*inst._generic))[1]
> >             except IndexError:  # empty GenericDict
> >                 return ()
> 
> That's... weird. Sometimes your instance will return a list, and sometimes a
> tuple. So if you write code expecting a list, it will suddenly and
> unexpectedly break when the instance is empty.
> 
> A better way of writing this is:
> 
>     def values(inst):
>         if isinstance(inst._generic, dict):
>             return inst._generic.values()
>         else:
>             return [t[1] for t in inst._generic]

You are right, I modified it to
`return [value for key, value in inst._generic]`
...which is more more intelligible than my original try/except/list
(zip)

Thanks, Thorsten



From ned at nedbatchelder.com  Sun Nov 20 07:54:09 2016
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Sun, 20 Nov 2016 04:54:09 -0800 (PST)
Subject: help on "from deen import *" vs. "import deen"
In-Reply-To: <3f64eee1-cfd8-a4a4-ff7f-f8f68505dea0@gmail.com>
References: <5d48ae5f-17f1-fca7-1239-d22c2950363c@gmail.com>
 <mailman.154.1479266093.30068.python-list@python.org>
 <c4c12dd3-7d2d-4a87-bdbc-0e752dbf5318@googlegroups.com>
 <ababec33-ad77-60f4-367f-aa89aad22230@gmail.com>
 <mailman.7.1479397099.20410.python-list@python.org>
 <023fdec4-3630-4030-a58b-24028dde4b41@googlegroups.com>
 <5ee9f8ce-e3a6-8163-02da-f5d928d219e1@gmail.com>
 <mailman.24.1479438177.20410.python-list@python.org>
 <90551ab1-c0b9-48db-85c3-c9487926b3f7@googlegroups.com>
 <kaiu2cdjgd8bu94tvhvve532ug3v8r3uf6@4ax.com>
 <mailman.40.1479495082.20410.python-list@python.org>
 <582fd63b$0$1598$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmroWO8iKnFRrGA_9yjKMfJdiuPpDhM0a3jxZ=QZztJHgA@mail.gmail.com>
 <mailman.48.1479538706.20410.python-list@python.org>
 <3f64eee1-cfd8-a4a4-ff7f-f8f68505dea0@gmail.com>
Message-ID: <23ddffb7-8a8b-4c0d-8eab-069367b7b5ed@googlegroups.com>

On Sunday, November 20, 2016 at 6:47:35 AM UTC-5, Bev in TX wrote:
> From the Python 3.5.2 docs:
> 
> 6.15. Evaluation order
> Python evaluates expressions from left to right. Notice that while 
> evaluating an assignment, the right-hand side is evaluated before the 
> left-hand side.
> 
> Thus, spam = eggs = cheese = obj is equivalent to:
> 
> spam = (eggs = (cheese = obj))
> 
> which is also equivalent to:
> 
> cheese = obj
> eggs = cheese
> spam = eggs

This is not right.  Python will evaluate the right-hand side, and then
assign that value to all of the names, from left to right.  So the end
result is equivalent to:

    spam = obj
    eggs = obj
    cheese = obj

The reason the doc quote above doesn't apply here is that assignment
is not an expression, but a statement of its own, described in 6.2:

"An assignment statement evaluates the expression list (remember
that this can be a single expression or a comma-separated list,
the latter yielding a tuple) and assigns the single resulting object
to each of the target lists, from left to right."

You can see this by disassembling some examples:

    >>> import dis
    >>> def f():
    ...     a = b = c = expr()
    ...
    >>> dis.dis(f)
    2           0 LOAD_GLOBAL              0 (expr)
                2 CALL_FUNCTION            0
                4 DUP_TOP
                6 STORE_FAST               0 (a)
                8 DUP_TOP
                10 STORE_FAST               1 (b)
                12 STORE_FAST               2 (c)
                14 LOAD_CONST               0 (None)
                16 RETURN_VALUE
    >>> def g():
    ...     d[a()] = d[b()] = d[c()] = expr()
    ...
    >>> dis.dis(g)
    2           0 LOAD_GLOBAL              0 (expr)
                2 CALL_FUNCTION            0
                4 DUP_TOP
                6 LOAD_GLOBAL              1 (d)
                8 LOAD_GLOBAL              2 (a)
                10 CALL_FUNCTION            0
                12 STORE_SUBSCR
                14 DUP_TOP
                16 LOAD_GLOBAL              1 (d)
                18 LOAD_GLOBAL              3 (b)
                20 CALL_FUNCTION            0
                22 STORE_SUBSCR
                24 LOAD_GLOBAL              1 (d)
                26 LOAD_GLOBAL              4 (c)
                28 CALL_FUNCTION            0
                30 STORE_SUBSCR
                32 LOAD_CONST               0 (None)
                34 RETURN_VALUE

Of course, it hardly ever matters, because how often does anyone
use multiple assignment, and even rarer, where the targets are
computed, and even rarer, where the order of their assignment
matters?

--Ned.


From thorsten at thorstenkampe.de  Sun Nov 20 08:12:02 2016
From: thorsten at thorstenkampe.de (Thorsten Kampe)
Date: Sun, 20 Nov 2016 14:12:02 +0100
Subject: Generic dictionary
References: <o0rqae$t10$1@blaine.gmane.org> <o0rr71$4qr$3@blaine.gmane.org>
 <o0rutu$sr4$1@blaine.gmane.org> <o0s4cm$hnq$2@blaine.gmane.org>
 <CAPTjJmqff2oH9-jo7Gs5Qguu8Lg0AZ_u5V6ZV-o+oShbQj3wPg@mail.gmail.com>
Message-ID: <o0s7g1$hnq$5@blaine.gmane.org>

* Chris Angelico (Sun, 20 Nov 2016 23:35:52 +1100)
> I see. So you want to be able to have something that looks and 
> feels
> like a dictionary, but uses a different way of looking things up.
> Makes reasonable sense, on the surface.
> 
> Before you go down that route, I strongly recommend reading up on
> exactly *why* a dictionary has the requirement of hashability, and
> what the consequences are of trying to look things up using lists as
> keys. You can easily experiment with it like this:
> 
> class HashableList(list):
>     def __hash__(self):
>         return hash(tuple(self))
> 
> Use those in your dict, rather than vanilla lists. Then you can mess
> around with the consequences of mutable dict keys.
> 
> Alternatively, switch from using lists to using tuples. They're
> hashable and immutable, thus avoiding the problems. If what you're
> trying to do is use multi-part dict keys, a tuple is far and away the
> best solution.

Chris, it's up to the consumer to decide which key/keyfunc to use. 
The lists as keys (that is first element in an tuple) are created on 
the fly and not used for anything. They are mutable but not mutated.

Mutating the list would be like mutating a list you're iterating 
through: just don't do it.

Thorsten



From rosuav at gmail.com  Sun Nov 20 08:28:48 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 21 Nov 2016 00:28:48 +1100
Subject: Generic dictionary
In-Reply-To: <o0s7g1$hnq$5@blaine.gmane.org>
References: <o0rqae$t10$1@blaine.gmane.org> <o0rr71$4qr$3@blaine.gmane.org>
 <o0rutu$sr4$1@blaine.gmane.org> <o0s4cm$hnq$2@blaine.gmane.org>
 <CAPTjJmqff2oH9-jo7Gs5Qguu8Lg0AZ_u5V6ZV-o+oShbQj3wPg@mail.gmail.com>
 <o0s7g1$hnq$5@blaine.gmane.org>
Message-ID: <CAPTjJmr5iYKAAYvt-r1cXsqGoX3uBJ0p_RE_d4N=stDku+TQ0A@mail.gmail.com>

On Mon, Nov 21, 2016 at 12:12 AM, Thorsten Kampe
<thorsten at thorstenkampe.de> wrote:
> Chris, it's up to the consumer to decide which key/keyfunc to use.
> The lists as keys (that is first element in an tuple) are created on
> the fly and not used for anything. They are mutable but not mutated.
>
> Mutating the list would be like mutating a list you're iterating
> through: just don't do it.

Then use tuples. Job done! :)

ChrisA


From countryone77 at gmail.com  Sun Nov 20 09:41:46 2016
From: countryone77 at gmail.com (Bev in TX)
Date: Sun, 20 Nov 2016 08:41:46 -0600
Subject: help on "from deen import *" vs. "import deen"
In-Reply-To: <3f64eee1-cfd8-a4a4-ff7f-f8f68505dea0@gmail.com>
References: <5d48ae5f-17f1-fca7-1239-d22c2950363c@gmail.com>
 <mailman.154.1479266093.30068.python-list@python.org>
 <c4c12dd3-7d2d-4a87-bdbc-0e752dbf5318@googlegroups.com>
 <ababec33-ad77-60f4-367f-aa89aad22230@gmail.com>
 <mailman.7.1479397099.20410.python-list@python.org>
 <023fdec4-3630-4030-a58b-24028dde4b41@googlegroups.com>
 <5ee9f8ce-e3a6-8163-02da-f5d928d219e1@gmail.com>
 <mailman.24.1479438177.20410.python-list@python.org>
 <90551ab1-c0b9-48db-85c3-c9487926b3f7@googlegroups.com>
 <kaiu2cdjgd8bu94tvhvve532ug3v8r3uf6@4ax.com>
 <mailman.40.1479495082.20410.python-list@python.org>
 <582fd63b$0$1598$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmroWO8iKnFRrGA_9yjKMfJdiuPpDhM0a3jxZ=QZztJHgA@mail.gmail.com>
 <mailman.48.1479538706.20410.python-list@python.org>
 <3f64eee1-cfd8-a4a4-ff7f-f8f68505dea0@gmail.com>
Message-ID: <o0scm8$7n7$1@dont-email.me>

Thanks to ChrisA and Ned for that clarification.

Bev in TX



From jfong at ms4.hinet.net  Sun Nov 20 20:32:40 2016
From: jfong at ms4.hinet.net (jfong at ms4.hinet.net)
Date: Sun, 20 Nov 2016 17:32:40 -0800 (PST)
Subject: help on "from deen import *" vs. "import deen"
In-Reply-To: <o0s4it$c8a$2@dont-email.me>
References: <cdc7bc72-d613-4b8b-a716-1e3514cfbb8e@googlegroups.com>
 <ca14fafb-2226-493e-ada5-e850a3c74f48@googlegroups.com>
 <e0478664-9610-4d83-8f58-583682d11fcd@googlegroups.com>
 <bb816fbf-1df7-522d-bab7-9f7a65deaef3@mrabarnett.plus.com>
 <mailman.147.1479180690.30068.python-list@python.org>
 <1a6f991d-53c1-4c79-b81b-e804b01be653@googlegroups.com>
 <144b0566-6738-bd37-cd1c-1585e70761d2@gmail.com>
 <9b6ad48d-ee65-bca6-6fa7-4176ddaae83e@lucidity.plus.com>
 <mailman.153.1479248175.30068.python-list@python.org>
 <o0s4it$c8a$2@dont-email.me>
Message-ID: <e5cf2ca2-8464-47ce-a158-4584c832791e@googlegroups.com>

Tristan B. Kildaire at 2016/11/20 8:23:37PM wrote:
> From deen import * imports all the things in deen but accessable with no 
> `deen.`

These "accessible" objects become read-only even if it's mutable. For immutable objects, you can't even create a new one in the deen's namespace.

> import deen imports all of deen but accessible via `deen.thing` 

:-)

--Jach


From bc at freeuk.com  Sun Nov 20 20:38:47 2016
From: bc at freeuk.com (BartC)
Date: Mon, 21 Nov 2016 01:38:47 +0000
Subject: Numpy slow at vector cross product?
In-Reply-To: <o0t22o$d9r$3@dont-email.me>
References: <o0t22o$d9r$3@dont-email.me>
Message-ID: <o0tj66$h42$1@dont-email.me>

On 20/11/2016 20:46, DFS wrote:
> import sys, time, numpy as np
> loops=int(sys.argv[1])
>
> x=np.array([1,2,3])
> y=np.array([4,5,6])
> start=time.clock()
> for i in range(loops):
>     np.cross(x,y)
> print "Numpy, %s loops: %.2g seconds" %(loops,time.clock()-start)
>
> x=[1,2,3]
> y=[4,5,6]
> z=[0,0,0]
> start=time.clock()
> for i in range(loops):
>     z[0]=x[1]*y[2]-x[2]*y[1];
>     z[1]=x[2]*y[0]-x[0]*y[2];
>     z[2]=x[0]*y[1]-x[1]*y[0];
> print "Calc, %s loops: %.2g seconds" %(loops,time.clock()-start)

I don't know why numpy is slow, but I can confirm similar results.

In fact if the workings are put into a function, then the difference is 
even more marked, with the normal calculation being 50% faster

Maybe numpy has extra overheads, and the arrays being operated on are 
very small, but even so, 30 times slower than CPython? (2.5 to 0.083 
seconds.)

-- 
Bartc






From steve+python at pearwood.info  Sun Nov 20 21:48:01 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Mon, 21 Nov 2016 13:48:01 +1100
Subject: Numpy slow at vector cross product?
References: <o0t22o$d9r$3@dont-email.me>
Message-ID: <58326062$0$22141$c3e8da3$5496439d@news.astraweb.com>

On Mon, 21 Nov 2016 07:46 am, DFS wrote:

> import sys, time, numpy as np
> loops=int(sys.argv[1])
> 
> x=np.array([1,2,3])
> y=np.array([4,5,6])
> start=time.clock()
> for i in range(loops):
>      np.cross(x,y)
> print "Numpy, %s loops: %.2g seconds" %(loops,time.clock()-start)

[...]
> $ python vector_cross.py
> Numpy, 100000 loops: 2.5 seconds
> Calc,  100000 loops: 0.13 seconds
> 
> 
> Did I do something wrong, or is numpy slow at this?

I can confirm similar results.

However, your code is not a great way of timing code. Timing code is *very*
difficult, and can be effected by many things, such as external processes,
CPU caches, even the function you use for getting the time. Much of the
time you are timing here will be in creating the range(loops) list,
especially if loops is big.

The best way to time small snippets of code is to use the timeit module.
Open a terminal or shell (*not* the Python interactive interpreter, the
operating system's shell: you should expect a $ or % prompt) and run timeit
from that. Copy and paste the following two commands into your shell
prompt:


python2.7 -m timeit --repeat 5 -s "import numpy as np" \
-s "x = np.array([1, 2, 3])" -s "y = np.array([4, 5, 6])" \
-- "np.cross(x, y)"


python2.7 -m timeit --repeat 5 -s "x = [1, 2, 3]" \
-s "y = [4, 5, 6]" -s "z = [0, 0, 0]" \
-- "z[0] = x[1]*y[2] - x[2]*y[1]; z[1] = x[2]*y[0] - \
x[0]*y[2]; z[2] = x[0]*y[1] - x[1]*y[0]"


The results I get are:

10000 loops, best of 5: 30 usec per loop

1000000 loops, best of 5: 1.23 usec per loop


So on my machine, np.cross() is about 25 times slower than multiplying by
hand.





-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From iivri.andre at gmail.com  Mon Nov 21 01:22:53 2016
From: iivri.andre at gmail.com (iivri.andre at gmail.com)
Date: Sun, 20 Nov 2016 22:22:53 -0800 (PST)
Subject: TemplateError
Message-ID: <4eaee64e-f104-4e6b-af24-9f0fe7063fc0@googlegroups.com>

THIS error is constantly showing up when I run my python script 
 eloiim:build iivri.andre$ python run.py 
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
[2016-11-21 01:15:26,561] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1988, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1641, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1544, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1639, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1625, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "run.py", line 10, in IsSelf
    return render_template('index.html', author=author, name=name)
  File "/usr/local/lib/python2.7/site-packages/flask/templating.py", line 133, in render_template
    return _render(ctx.app.jinja_env.get_or_select_template(template_name_or_list),
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 851, in get_or_select_template
    return self.get_template(template_name_or_list, parent, globals)
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 812, in get_template
    return self._load_template(name, self.make_globals(globals))
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 774, in _load_template
    cache_key = self.loader.get_source(self, name)[1]
  File "/usr/local/lib/python2.7/site-packages/flask/templating.py", line 57, in get_source
    return self._get_source_fast(environment, template)
  File "/usr/local/lib/python2.7/site-packages/flask/templating.py", line 85, in _get_source_fast
    raise TemplateNotFound(template)
TemplateNotFound: index.html

THIS is the code 

from flask import Flask, render_template
app = Flask(__name__)

@app.route('/')
# THE @ is a decorator that is used to 'augment' function definitions
# Flask uses 'route()' to indicate that if the browser requests the address '/' (the default | home address), then our app shoud 'route' that request to this 'IsSelf' function
def IsSelf() :
    author = "raphael James "
    name = "Jaden iivii"
    return render_template('index.html', author=author, name=name)

if __name__ == "__main__" :
    app.run()


How do I fix this? 


From rosuav at gmail.com  Mon Nov 21 01:44:38 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 21 Nov 2016 17:44:38 +1100
Subject: TemplateError
In-Reply-To: <4eaee64e-f104-4e6b-af24-9f0fe7063fc0@googlegroups.com>
References: <4eaee64e-f104-4e6b-af24-9f0fe7063fc0@googlegroups.com>
Message-ID: <CAPTjJmqQdpZ4siWVeqPX9xUhzWK-9uaJRwm6bgKyHdRVAgVhUQ@mail.gmail.com>

On Mon, Nov 21, 2016 at 5:22 PM,  <iivri.andre at gmail.com> wrote:
> TemplateNotFound: index.html
>
>     return render_template('index.html', author=author, name=name)

The render_template function looks for a directory called "templates"
and a file in that of the given name. So you'll need to have
"templates/index.html" available. Possibly you have index.html in the
wrong place?

ChrisA


From steve+comp.lang.python at pearwood.info  Mon Nov 21 02:09:51 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Mon, 21 Nov 2016 18:09:51 +1100
Subject: Clean way to return error codes
Message-ID: <58329dc1$0$2846$c3e8da3$76491128@news.astraweb.com>

I have a script that can be broken up into four subtasks. If any of those 
subtasks fail, I wish to exit with a different exit code and error.

Assume that the script is going to be run by system administrators who know no 
Python and are terrified of tracebacks, and that I'm logging the full traceback 
elsewhere (not shown).

I have something like this:


try:
    begin()
except BeginError:
    print("error in begin")
    sys.exit(3)

try:
    cur = get_cur()
except FooError:
    print("failed to get cur")
    sys.exit(17)

try:
    result = process(cur)
    print(result)
except FooError, BarError:
    print("error in processing")
    sys.exit(12)

try:
    cleanup()
except BazError:
    print("cleanup failed")
    sys.exit(8)



It's not awful, but I don't really like the look of all those try...except 
blocks. Is there something cleaner I can do, or do I just have to suck it up?




-- 
Steven
299792.458 km/s ? not just a good idea, it?s the law!



From vincent.vande.vyvre at telenet.be  Mon Nov 21 02:29:50 2016
From: vincent.vande.vyvre at telenet.be (Vincent Vande Vyvre)
Date: Mon, 21 Nov 2016 08:29:50 +0100
Subject: TemplateError
In-Reply-To: <4eaee64e-f104-4e6b-af24-9f0fe7063fc0@googlegroups.com>
References: <4eaee64e-f104-4e6b-af24-9f0fe7063fc0@googlegroups.com>
Message-ID: <78857d7e-694f-8e80-b991-2f7a175e3763@telenet.be>

Le 21/11/2016 ? 07:22, iivri.andre at gmail.com a ?crit :
> THIS error is constantly showing up when I run my python script
>   eloiim:build iivri.andre$ python run.py
>   * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
> [2016-11-21 01:15:26,561] ERROR in app: Exception on / [GET]
> Traceback (most recent call last):
>    File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1988, in wsgi_app
>      response = self.full_dispatch_request()
>    File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1641, in full_dispatch_request
>      rv = self.handle_user_exception(e)
>    File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1544, in handle_user_exception
>      reraise(exc_type, exc_value, tb)
>    File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1639, in full_dispatch_request
>      rv = self.dispatch_request()
>    File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1625, in dispatch_request
>      return self.view_functions[rule.endpoint](**req.view_args)
>    File "run.py", line 10, in IsSelf
>      return render_template('index.html', author=author, name=name)
>    File "/usr/local/lib/python2.7/site-packages/flask/templating.py", line 133, in render_template
>      return _render(ctx.app.jinja_env.get_or_select_template(template_name_or_list),
>    File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 851, in get_or_select_template
>      return self.get_template(template_name_or_list, parent, globals)
>    File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 812, in get_template
>      return self._load_template(name, self.make_globals(globals))
>    File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 774, in _load_template
>      cache_key = self.loader.get_source(self, name)[1]
>    File "/usr/local/lib/python2.7/site-packages/flask/templating.py", line 57, in get_source
>      return self._get_source_fast(environment, template)
>    File "/usr/local/lib/python2.7/site-packages/flask/templating.py", line 85, in _get_source_fast
>      raise TemplateNotFound(template)
> TemplateNotFound: index.html
>
> THIS is the code
>
> from flask import Flask, render_template
> app = Flask(__name__)
>
> @app.route('/')
> # THE @ is a decorator that is used to 'augment' function definitions
> # Flask uses 'route()' to indicate that if the browser requests the address '/' (the default | home address), then our app shoud 'route' that request to this 'IsSelf' function
> def IsSelf() :
>      author = "raphael James "
>      name = "Jaden iivii"
>      return render_template('index.html', author=author, name=name)
>
> if __name__ == "__main__" :
>      app.run()
>
>
> How do I fix this?

Flask don't know where are your templates.

I use this:

WEBDIR = os.path.dirname(os.path.abspath('__files__'))
PAGESDIR = os.path.join(os.path.dirname(WEBDIR), 'somewhere/templates')
STATICDIR = os.path.join(os.path.dirname(WEBDIR), 'somewhere/static')
...

app = Flask(__name__, template_folder=PAGESDIR, static_folder=STATICDIR)
...

-- 
Vincent V.V.
Oqapy <http://www.oqapy.eu> . python3-exiv2 
<http://www.py3exiv2.tuxfamily.org/> . Qarte 
<https://launchpad.net/qarte> . PaQager <https://launchpad.net/paqager>


From jussi.piitulainen at helsinki.fi  Mon Nov 21 02:39:35 2016
From: jussi.piitulainen at helsinki.fi (Jussi Piitulainen)
Date: Mon, 21 Nov 2016 09:39:35 +0200
Subject: Clean way to return error codes
References: <58329dc1$0$2846$c3e8da3$76491128@news.astraweb.com>
Message-ID: <lf5wpfx5ljc.fsf@ling.helsinki.fi>

Steven D'Aprano writes:

> I have a script that can be broken up into four subtasks. If any of
> those subtasks fail, I wish to exit with a different exit code and
> error.
>
> Assume that the script is going to be run by system administrators who
> know no Python and are terrified of tracebacks, and that I'm logging
> the full traceback elsewhere (not shown).
>
> I have something like this:
>
>
> try:
>     begin()
> except BeginError:
>     print("error in begin")
>     sys.exit(3)
>
> try:
>     cur = get_cur()
> except FooError:
>     print("failed to get cur")
>     sys.exit(17)
>
> try:
>     result = process(cur)
>     print(result)
> except FooError, BarError:
>     print("error in processing")
>     sys.exit(12)
>
> try:
>     cleanup()
> except BazError:
>     print("cleanup failed")
>     sys.exit(8)
>
>
>
> It's not awful, but I don't really like the look of all those
> try...except blocks. Is there something cleaner I can do, or do I just
> have to suck it up?

Have the exception objects carry the message and the exit code?

try:
    begin()
    cur = get_cur()
    result = process(cur)
    print(result)
    cleanup()
except (BeginError, FooError, BarError, BazError) as exn:
    print("Steven's script:", message(exn))
    sys.exit(code(exn))


From ben+python at benfinney.id.au  Mon Nov 21 03:01:17 2016
From: ben+python at benfinney.id.au (Ben Finney)
Date: Mon, 21 Nov 2016 19:01:17 +1100
Subject: Clean way to return error codes
References: <58329dc1$0$2846$c3e8da3$76491128@news.astraweb.com>
Message-ID: <85lgwd45yq.fsf@benfinney.id.au>

Steven D'Aprano <steve+comp.lang.python at pearwood.info> writes:

> I have a script that can be broken up into four subtasks. If any of those 
> subtasks fail, I wish to exit with a different exit code and error.
>
> Assume that the script is going to be run by system administrators who
> know no Python and are terrified of tracebacks, and that I'm logging
> the full traceback elsewhere (not shown).

The first improvement I'd make (and you likely already know this one,
but for the benefit of later readers):

    try:
        begin()
    except FooError as exc:
        print("failed to get cur: {message}".format(message=exc))
        raise SystemExit(17) from exc

That preserves the full traceback for anything that instruments that
program to try to debug what went wrong. It will also emit the message
from the underlying problem, for the user who only sees the program
output.


The next improvement I'd make is to use the ?strategy? pattern, and come
up with some common key that determines what exit status you want. Maybe
the key is a tuple of (function, exception):

    exit_status_by_problem_key = {
            (begin, BeginError): 3,
            (get_cur, FooError): 17,
            (process, FooError): 12,
            (process, BarError): 12,
            (cleanup, BazError): 8,
            }

Or you might need to define the strategy as (message_template,
exit_status):

    exit_status_by_problem_key = {
            (begin, BeginError): ("error in begin: {message}", 3),
            (get_cur, FooError): ("failed to get cur: {message}", 17),
            (process, FooError): ("error in processing: {message}", 12),
            (process, BarError): ("error in processing: {message}", 12),
            (cleanup, BazError): ("cleanup failed: {message}", 8),
            }

-- 
 \      ?God forbid that any book should be banned. The practice is as |
  `\                  indefensible as infanticide.? ?Dame Rebecca West |
_o__)                                                                  |
Ben Finney



From steve+comp.lang.python at pearwood.info  Mon Nov 21 03:06:07 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Mon, 21 Nov 2016 19:06:07 +1100
Subject: Clean way to return error codes
References: <58329dc1$0$2846$c3e8da3$76491128@news.astraweb.com>
 <lf5wpfx5ljc.fsf@ling.helsinki.fi>
Message-ID: <5832aaef$0$11097$c3e8da3@news.astraweb.com>

On Monday 21 November 2016 18:39, Jussi Piitulainen wrote:

> Steven D'Aprano writes:
[...]
>> It's not awful, but I don't really like the look of all those
>> try...except blocks. Is there something cleaner I can do, or do I just
>> have to suck it up?
> 
> Have the exception objects carry the message and the exit code?
> 
> try:
>     begin()
>     cur = get_cur()
>     result = process(cur)
>     print(result)
>     cleanup()
> except (BeginError, FooError, BarError, BazError) as exn:
>     print("Steven's script:", message(exn))
>     sys.exit(code(exn))

Oooh, nice!



-- 
Steven
299792.458 km/s ? not just a good idea, it?s the law!



From steve+comp.lang.python at pearwood.info  Mon Nov 21 03:08:25 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Mon, 21 Nov 2016 19:08:25 +1100
Subject: Clean way to return error codes
References: <58329dc1$0$2846$c3e8da3$76491128@news.astraweb.com>
 <85lgwd45yq.fsf@benfinney.id.au>
 <mailman.14.1479715297.3146.python-list@python.org>
Message-ID: <5832ab7a$0$11097$c3e8da3@news.astraweb.com>

On Monday 21 November 2016 19:01, Ben Finney wrote:

[...]
> The next improvement I'd make is to use the ?strategy? pattern, and come
> up with some common key that determines what exit status you want. Maybe
> the key is a tuple of (function, exception):
> 
>     exit_status_by_problem_key = {
>             (begin, BeginError): 3,
>             (get_cur, FooError): 17,
>             (process, FooError): 12,
>             (process, BarError): 12,
>             (cleanup, BazError): 8,
>             }

Indeed. Probably not worth the overhead in extra complexity for just four sub-
tasks, but worth keeping in mind.


-- 
Steven
299792.458 km/s ? not just a good idea, it?s the law!



From steve+comp.lang.python at pearwood.info  Mon Nov 21 03:12:46 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Mon, 21 Nov 2016 19:12:46 +1100
Subject: Setting the exit status from sys.excepthook
Message-ID: <5832ac7e$0$11097$c3e8da3@news.astraweb.com>

I have script with an exception handler that takes care of writing the 
traceback to syslog, and I set it as the global exceptionhook:

sys.excepthook = my_error_handler


When my script raises, my_error_handler is called, as expected, and the process 
exits with status 1.

How can I change the exit status to another value, but only for exceptions 
handled by my_error_handler?



-- 
Steven
299792.458 km/s ? not just a good idea, it?s the law!



From rosuav at gmail.com  Mon Nov 21 03:18:08 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 21 Nov 2016 19:18:08 +1100
Subject: Clean way to return error codes
In-Reply-To: <58329dc1$0$2846$c3e8da3$76491128@news.astraweb.com>
References: <58329dc1$0$2846$c3e8da3$76491128@news.astraweb.com>
Message-ID: <CAPTjJmpEssT_g8+V2dm-RrX7Cf6P8pVLaKKMS4dMFFE=pZZGkg@mail.gmail.com>

On Mon, Nov 21, 2016 at 6:09 PM, Steven D'Aprano
<steve+comp.lang.python at pearwood.info> wrote:
> try:
>     begin()
> except BeginError:
>     print("error in begin")
>     sys.exit(3)

Do you control the errors that are getting thrown?

class BeginExit(SystemExit, BeginError): pass

It'll behave like SystemExit, but still be catchable as BeginError.
(Or if BeginError isn't used anywhere else, it can itself be redefined
to inherit from SystemExit.)

ChrisA


From bob.martin at excite.com  Mon Nov 21 03:29:35 2016
From: bob.martin at excite.com (Bob Martin)
Date: Mon, 21 Nov 2016 08:29:35 GMT
Subject: Guido? Where are you?
References: <o0s4l0$c8a$3@dont-email.me> <1853932.OBFZWjSADL@PointedEars.de>
Message-ID: <e9fpjfFl65tU1@mid.individual.net>

in 767657 20161121 041134 Thomas 'PointedEars' Lahn <PointedEars at web.de> wrote:
>Tristan B. Kildaire wrote:
>
>> Is Guido active on this newsgroup.
>
>That is not even a question.
>
>> Sorry for the off-topic ness.
>
>There is no excuse for (such) stupidity.

Stop posting then.

>
><http://catb.org/esr/faqs/smart-questions.html>
>


From ethan at stoneleaf.us  Mon Nov 21 03:53:33 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 21 Nov 2016 00:53:33 -0800
Subject: Guido? Where are you?
In-Reply-To: <e9fpjfFl65tU1@mid.individual.net>
References: <o0s4l0$c8a$3@dont-email.me> <1853932.OBFZWjSADL@PointedEars.de>
 <e9fpjfFl65tU1@mid.individual.net>
Message-ID: <5832B60D.906@stoneleaf.us>

For those new to the list:

Thomas 'PointedEars' Lahn <PointedEars at web.de>

has been banned from this mailing list for (at least) the rest of this year.  Unfortunately, we do not have any control over the comp.lang.python newsgroup.  If you access Python List from the newsgroup you may want to add him to your personal kill-file so you don't have to see his posts.

--
~Ethan~


From __peter__ at web.de  Mon Nov 21 04:13:24 2016
From: __peter__ at web.de (Peter Otten)
Date: Mon, 21 Nov 2016 10:13:24 +0100
Subject: Clean way to return error codes
References: <58329dc1$0$2846$c3e8da3$76491128@news.astraweb.com>
Message-ID: <o0udrh$tmf$1@blaine.gmane.org>

Steven D'Aprano wrote:

> I have a script that can be broken up into four subtasks. If any of those
> subtasks fail, I wish to exit with a different exit code and error.
> 
> Assume that the script is going to be run by system administrators who
> know no Python and are terrified of tracebacks, and that I'm logging the
> full traceback elsewhere (not shown).
> 
> I have something like this:
> 
> 
> try:
>     begin()
> except BeginError:
>     print("error in begin")
>     sys.exit(3)
> 
> try:
>     cur = get_cur()
> except FooError:
>     print("failed to get cur")
>     sys.exit(17)
> 
> try:
>     result = process(cur)
>     print(result)
> except FooError, BarError:
>     print("error in processing")
>     sys.exit(12)
> 
> try:
>     cleanup()
> except BazError:
>     print("cleanup failed")
>     sys.exit(8)
> 
> 
> 
> It's not awful, but I don't really like the look of all those try...except
> blocks. Is there something cleaner I can do, or do I just have to suck it
> up?

def run():
    yield BeginError, "error in begin", 3
    begin()

    yield FooError, "failed to get cur", 17
    cur = get_cur()

    yield (FooError, BarError), "error in processing", 12
    result = process(cur)
    print(result)

    yield BazError, "cleanup failed", 8
    cleanup()

try:
    for Errors, message, exitcode in run():
        pass
except Errors:
    print(message)
    sys.exit(exitcode)




From bc at freeuk.com  Mon Nov 21 07:09:31 2016
From: bc at freeuk.com (BartC)
Date: Mon, 21 Nov 2016 12:09:31 +0000
Subject: Numpy slow at vector cross product?
In-Reply-To: <58326062$0$22141$c3e8da3$5496439d@news.astraweb.com>
References: <o0t22o$d9r$3@dont-email.me>
 <58326062$0$22141$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o0uo4r$gno$1@dont-email.me>

On 21/11/2016 02:48, Steve D'Aprano wrote:
> On Mon, 21 Nov 2016 07:46 am, DFS wrote:

>> start=time.clock()
>> for i in range(loops):
>>      np.cross(x,y)
>> print "Numpy, %s loops: %.2g seconds" %(loops,time.clock()-start)

> However, your code is not a great way of timing code. Timing code is *very*
> difficult, and can be effected by many things, such as external processes,
> CPU caches, even the function you use for getting the time. Much of the
> time you are timing here will be in creating the range(loops) list,
> especially if loops is big.

But both loops are the same size. And that overhead can quickly be 
disposed of by measuring empty loops in both cases. (On my machine, 
about 0.006/7 seconds for loops of 100,000.)

> The best way to time small snippets of code is to use the timeit module.
> Open a terminal or shell (*not* the Python interactive interpreter, the
> operating system's shell: you should expect a $ or % prompt) and run timeit
> from that. Copy and paste the following two commands into your shell
> prompt:
>
>
> python2.7 -m timeit --repeat 5 -s "import numpy as np" \
> -s "x = np.array([1, 2, 3])" -s "y = np.array([4, 5, 6])" \
> -- "np.cross(x, y)"
>
>
> python2.7 -m timeit --repeat 5 -s "x = [1, 2, 3]" \
> -s "y = [4, 5, 6]" -s "z = [0, 0, 0]" \
> -- "z[0] = x[1]*y[2] - x[2]*y[1]; z[1] = x[2]*y[0] - \
> x[0]*y[2]; z[2] = x[0]*y[1] - x[1]*y[0]"

Yes, I can see that typing all the code out again, and remembering all 
those options and putting -s, -- and \ in all the right places, is a 
much better way of doing it! Not error prone at all.

-- 
bartc


From vek.m1234 at gmail.com  Mon Nov 21 07:15:12 2016
From: vek.m1234 at gmail.com (Veek M)
Date: Mon, 21 Nov 2016 17:45:12 +0530
Subject: What is the difference between class Foo(): and class Date(object):
Message-ID: <o0uoff$ht5$1@dont-email.me>

>>> class Foo():
...  pass
... 
>>> class Bar(Foo):
...  pass
... 
>>> b = Bar()
>>> type(b)
<type 'instance'>

>>> class Date(object):
...  pass
... 
>>> class EuroDate(Date):
...  pass
... 
>>> x = EuroDate()
>>> type(x)
<class '__main__.EuroDate'>


What is going on here? Shouldn't x = EuroDate();  type(x) give 
'instance'?? Why is 'b' an 'instance' and 'x' EuroDate?
Why isn't 'b' Bar?

----
Guhh! (I am reading @classmethods from beazley - i know i have two open 
threads but I'll get to that - will require even more reading)




From __peter__ at web.de  Mon Nov 21 07:44:25 2016
From: __peter__ at web.de (Peter Otten)
Date: Mon, 21 Nov 2016 13:44:25 +0100
Subject: Numpy slow at vector cross product?
References: <o0t22o$d9r$3@dont-email.me>
 <58326062$0$22141$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o0uq77$cjf$1@blaine.gmane.org>

Steve D'Aprano wrote:

> On Mon, 21 Nov 2016 07:46 am, DFS wrote:
> 
>> import sys, time, numpy as np
>> loops=int(sys.argv[1])
>> 
>> x=np.array([1,2,3])
>> y=np.array([4,5,6])
>> start=time.clock()
>> for i in range(loops):
>>      np.cross(x,y)
>> print "Numpy, %s loops: %.2g seconds" %(loops,time.clock()-start)
> 
> [...]
>> $ python vector_cross.py
>> Numpy, 100000 loops: 2.5 seconds
>> Calc,  100000 loops: 0.13 seconds
>> 
>> 
>> Did I do something wrong, or is numpy slow at this?
> 
> I can confirm similar results.
> 
> However, your code is not a great way of timing code. Timing code is
> *very* difficult, and can be effected by many things, such as external
> processes, CPU caches, even the function you use for getting the time.
> Much of the time you are timing here will be in creating the range(loops)
> list, especially if loops is big.
> 
> The best way to time small snippets of code is to use the timeit module.
> Open a terminal or shell (*not* the Python interactive interpreter, the
> operating system's shell: you should expect a $ or % prompt) and run
> timeit from that. Copy and paste the following two commands into your
> shell prompt:
> 
> 
> python2.7 -m timeit --repeat 5 -s "import numpy as np" \
> -s "x = np.array([1, 2, 3])" -s "y = np.array([4, 5, 6])" \
> -- "np.cross(x, y)"
> 
> 
> python2.7 -m timeit --repeat 5 -s "x = [1, 2, 3]" \
> -s "y = [4, 5, 6]" -s "z = [0, 0, 0]" \
> -- "z[0] = x[1]*y[2] - x[2]*y[1]; z[1] = x[2]*y[0] - \
> x[0]*y[2]; z[2] = x[0]*y[1] - x[1]*y[0]"
> 
> 
> The results I get are:
> 
> 10000 loops, best of 5: 30 usec per loop
> 
> 1000000 loops, best of 5: 1.23 usec per loop
> 
> 
> So on my machine, np.cross() is about 25 times slower than multiplying by
> hand.

After a look into the source this is no longer a big surprise (numpy 1.8.2):

    if axis is not None:
        axisa, axisb, axisc=(axis,)*3
    a = asarray(a).swapaxes(axisa, 0)
    b = asarray(b).swapaxes(axisb, 0)
    msg = "incompatible dimensions for cross product\n"\
          "(dimension must be 2 or 3)"
    if (a.shape[0] not in [2, 3]) or (b.shape[0] not in [2, 3]):
        raise ValueError(msg)
    if a.shape[0] == 2:
        if (b.shape[0] == 2):
            cp = a[0]*b[1] - a[1]*b[0]
            if cp.ndim == 0:
                return cp
            else:
                return cp.swapaxes(0, axisc)
        else:
            x = a[1]*b[2]
            y = -a[0]*b[2]
            z = a[0]*b[1] - a[1]*b[0]
    elif a.shape[0] == 3:
        if (b.shape[0] == 3):
            x = a[1]*b[2] - a[2]*b[1]
            y = a[2]*b[0] - a[0]*b[2]
            z = a[0]*b[1] - a[1]*b[0]
        else:
            x = -a[2]*b[1]
            y = a[2]*b[0]
            z = a[0]*b[1] - a[1]*b[0]
    cp = array([x, y, z])
    if cp.ndim == 1:
        return cp
    else:
        return cp.swapaxes(0, axisc)

The situation may be different when you process vectors in bulk, i. e. 
instead of [cross(a, bb) for bb in b] just say cross(a, b).




From __peter__ at web.de  Mon Nov 21 07:57:36 2016
From: __peter__ at web.de (Peter Otten)
Date: Mon, 21 Nov 2016 13:57:36 +0100
Subject: Setting the exit status from sys.excepthook
References: <5832ac7e$0$11097$c3e8da3@news.astraweb.com>
Message-ID: <o0uqvu$lpu$1@blaine.gmane.org>

Steven D'Aprano wrote:

> I have script with an exception handler that takes care of writing the
> traceback to syslog, and I set it as the global exceptionhook:
> 
> sys.excepthook = my_error_handler
> 
> 
> When my script raises, my_error_handler is called, as expected, and the
> process exits with status 1.
> 
> How can I change the exit status to another value, but only for exceptions
> handled by my_error_handler?

Why not just put

try:
   ...
except:
   ...

around the main function?

That said, it looks like you can exit() from the errorhandler:

$ cat bend_exit.py
import sys

def my_error_handler(etype, exception, traceback):
    print("unhandled", exception)
    sys.exit(2)

sys.excepthook = my_error_handler

1/int(sys.argv[1])
print("bye")
$ python3 bend_exit.py 0; echo $?
unhandled division by zero
2
$ python3 bend_exit.py 1; echo $?
bye
0

If "bad things" can happen I don't know...



From __peter__ at web.de  Mon Nov 21 08:01:27 2016
From: __peter__ at web.de (Peter Otten)
Date: Mon, 21 Nov 2016 14:01:27 +0100
Subject: Generic dictionary
References: <o0rqae$t10$1@blaine.gmane.org> <o0rr71$4qr$3@blaine.gmane.org>
 <o0s412$hnq$1@blaine.gmane.org>
Message-ID: <o0ur73$lpu$2@blaine.gmane.org>

Thorsten Kampe wrote:

>> def GenericDict(dict_or_items):
>>     if isinstance(dict_or_items, dict):
>>         return dict(dict_or_items)
>>     else:
>>         return SimpleGenericDictWithOnlyTheFalseBranchesImplemented(
>>             dict_or_items
>>         )
> 
> That would be a kind of factory function for the class?

Yes, one if...else to pick a dedicated class instead of one if...else per 
method in the all-things-to-all-people class.




From info at egenix.com  Mon Nov 21 08:40:19 2016
From: info at egenix.com (eGenix Team: M.-A. Lemburg)
Date: Mon, 21 Nov 2016 14:40:19 +0100
Subject: ANN: eGenix PyRun - One file Python Runtime 2.2.3
Message-ID: <5832F943.4090206@egenix.com>

________________________________________________________________________

ANNOUNCING

		eGenix PyRun - One file Python Runtime

			    Version 2.2.3


       An easy-to-use single file relocatable Python run-time -
	  available for Linux, Mac OS X and Unix platforms,
	      with support for Python 2.6, 2.7, 3.4 and
			* also for Python 3.5 *


This announcement is also available on our web-site for online reading:
http://www.egenix.com/company/news/eGenix-PyRun-2.2.3-GA.html

________________________________________________________________________

INTRODUCTION

eGenix PyRun is our open source, one file, no installation version of
Python, making the distribution of a Python interpreter to run based
scripts and applications to Unix based systems as simple as copying a
single file.

eGenix PyRun's executable only needs 11MB for Python 2 and 13MB for
Python 3, but still supports most Python application and scripts - and
it can be compressed to just 3-4MB using upx, if needed.

Compared to a regular Python installation of typically 100MB on disk,
eGenix PyRun is ideal for applications and scripts that need to be
distributed to several target machines, client installations or
customers.

It makes "installing" Python on a Unix based system as simple as
copying a single file.

eGenix has been using eGenix PyRun internally in the mxODBC Connect
Server product since 2008 with great success and decided to make it
available as a stand-alone open-source product.

We provide both the source archive to build your own eGenix PyRun, as
well as pre-compiled binaries for Linux, FreeBSD and Mac OS X, as 32-
and 64-bit versions. The binaries can be downloaded manually, or you
can let our automatic install script install-pyrun take care of the
installation: ./install-pyrun dir and you're done.

Please see the product page for more details:

    http://www.egenix.com/products/python/PyRun/

________________________________________________________________________

NEWS

This minor level release of eGenix PyRun comes with the following
enhancements:

Enhancements / Changes
----------------------

 * Removed lzma module from PyRun for Python 3.x again, since this
   caused too many issues with incompatible/missing libzma.so
   references. The module is still being built as optional add-on and
   can be used if the necessary libs are available, but it will no
   longer prevent PyRun from working altogether.

install-pyrun Quick Install Enhancements
---------------------------------------------

eGenix PyRun includes a shell script called install-pyrun, which
greatly simplifies installation of PyRun. It works much like the
virtualenv shell script used for creating new virtual environments
(except that there's nothing virtual about PyRun environments).

    https://downloads.egenix.com/python/install-pyrun

With the script, an eGenix PyRun installation is as simple as running:

    ./install-pyrun targetdir

This will automatically detect the platform, download and install the
right pyrun version into targetdir.

We have updated this script since the last release:

 * Updated install-pyrun to default to eGenix PyRun 2.2.3 and its
   feature set.

For a complete list of changes, please see the eGenix PyRun Changelog:

    http://www.egenix.com/products/python/PyRun/changelog.html

________________________________________________________________________

LICENSE

eGenix PyRun is distributed under the eGenix.com Public License 1.1.0
which is an Open Source license similar to the Python license. You can
use eGenix PyRun in both commercial and non-commercial settings
without fee or charge.

Please see our license page for more details:

    http://www.egenix.com/products/python/PyRun/license.html

The package comes with full source code.

________________________________________________________________________

DOWNLOADS

The download archives and instructions for installing eGenix PyRun can
be found at:

    http://www.egenix.com/products/python/PyRun/

As always, we are providing pre-built binaries for all common
platforms: Windows 32/64-bit, Linux 32/64-bit, FreeBSD 32/64-bit, Mac
OS X 32/64-bit. Source code archives are available for installation on
other platforms, such as Solaris, AIX, HP-UX, etc.

_______________________________________________________________________

SUPPORT

Commercial support for this product is available from eGenix.com.
Please see

    http://www.egenix.com/services/support/

for details about our support offerings.

________________________________________________________________________

MORE INFORMATION

For more information about eGenix PyRun, licensing and download
instructions, please visit our web-site:

    http://www.egenix.com/products/python/PyRun/


About eGenix (http://www.egenix.com/):

    eGenix is a Python software project, consulting and product
    company delivering expert services and professional quality
    products for companies, Python users and developers. We specialize
    in database driven applications, large scale software designs and
    integration.

Enjoy,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Nov 21 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/



From eryksun at gmail.com  Mon Nov 21 09:02:13 2016
From: eryksun at gmail.com (eryk sun)
Date: Mon, 21 Nov 2016 14:02:13 +0000
Subject: Numpy slow at vector cross product?
In-Reply-To: <o0tj66$h42$1@dont-email.me>
References: <o0t22o$d9r$3@dont-email.me> <o0tj66$h42$1@dont-email.me>
Message-ID: <CACL+1atxQmgbiAZx_-601BVb17AWutHKQuFkrndR7d5xH-EoDw@mail.gmail.com>

On Mon, Nov 21, 2016 at 1:38 AM, BartC <bc at freeuk.com> wrote:
> On 20/11/2016 20:46, DFS wrote:
>>
>> import sys, time, numpy as np
>> loops=int(sys.argv[1])
>>
>> x=np.array([1,2,3])
>> y=np.array([4,5,6])
>> start=time.clock()

In Unix, time.clock doesn't measure wall-clock time, but rather an
approximation to the CPU time used by the current process. On the
other hand, time.time calls gettimeofday, if available, which has a
resolution of 1 microsecond. Python 2 timing tests should use
time.time on Unix.

In Windows, time.time calls GetSystemTimeAsFileTime, which has a
default resolution of only about 15 ms, adjustable down to about 1 ms.
On other hand, time.clock calls QueryPerformanceCounter, which has a
resolution of about 100 nanoseconds. Python 2 timing tests should use
time.clock on Windows.

In Python 3.3+, timing tests should use time.perf_counter. In Linux
this calls clock_gettime using a monotonic clock with a resolution of
1 nanosecond, and in Windows it calls QueryPerformanceCounter.

In any case, timeit.default_timer selects the best function to call
for a given platform.


From steve+python at pearwood.info  Mon Nov 21 09:13:28 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Tue, 22 Nov 2016 01:13:28 +1100
Subject: What is the difference between class Foo(): and class
 Date(object):
References: <o0uoff$ht5$1@dont-email.me>
Message-ID: <58330109$0$1609$c3e8da3$5496439d@news.astraweb.com>

On Mon, 21 Nov 2016 11:15 pm, Veek M wrote:

>>>> class Foo():
> ...  pass
> ...
>>>> class Bar(Foo):
> ...  pass
> ...
>>>> b = Bar()
>>>> type(b)
> <type 'instance'>
[...] 

> What is going on here? Shouldn't x = EuroDate();  type(x) give
> 'instance'?? Why is 'b' an 'instance' and 'x' EuroDate?
> Why isn't 'b' Bar?


It looks like you are running Python 2, and have stumbled across an
annoyance from the earliest days of Python: the "classic", or "old-style",
class.

Before Python 2.2, custom classes and built-in types like int, float, dict
and list were different. You have just discovered one of the ways they were
different: instances of custom classes all had the same type, even if the
class was different:

# Python 2

py> class Dog:
...     pass
...
py> class Cat:
...     pass
...
py> lassie = Dog()
py> garfield = Cat()
py> type(lassie) is type(garfield)
True
py> type(lassie)
<type 'instance'>


This is just the most obvious difference between "classic classes" and
types. Some of the other differences:

- The method resolution order (MRO) is different: the classic class
  MRO is buggy for diamond-shaped multiple inheritance, and special
  dunder methods like __eq__ are resolved slightly differently.

- super, properties, class methods and static methods don't work for
  classic classes.

- The metaclass of classic classes is different:

py> type(Dog)
<type 'classobj'>
py> type(float)
<type 'type'>

- Attribute lookup for classic classes is slightly different; in
  particular, the special __getattribute__ method doesn't work.


In Python 2.2, the built-in types (list, dict, float etc) were unified with
the class mechanism, but for backwards compatibility the old-style classes
had to be left in. So Python had two class mechanisms:

- "New-style classes", or types, inherit from object, or some 
  other built-in type, and support properties, etc.

- "Old-style classes", don't inherit from object, don't support
  properties etc.


So in Python 2, when you write:

    class Foo:

or

    class Foo():


you get an old-style class. But when you inherit from object, you get a
new-style class. Classic classes are an obsolete feature from Python 2.
They are removed in Python 3, and things are much simpler. In Python 3, it
doesn't matter whether you write:

    class Foo:
    class Foo():
    class Foo(object):

the result is the same: a new-style class, or type.

The best thing to do in Python 2 is to always, without exception, write

    class Foo(object):

to define your base classes. That will ensure that property, super,
classmethod, staticmethod, __getattribute__, etc. will all work correctly,
and you will avoid the surprises of classic classes.



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Mon Nov 21 09:50:16 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Tue, 22 Nov 2016 01:50:16 +1100
Subject: Numpy slow at vector cross product?
References: <o0t22o$d9r$3@dont-email.me>
 <58326062$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <o0uo4r$gno$1@dont-email.me>
Message-ID: <583309a9$0$1596$c3e8da3$5496439d@news.astraweb.com>

On Mon, 21 Nov 2016 11:09 pm, BartC wrote:

> On 21/11/2016 02:48, Steve D'Aprano wrote:
[...]
>> However, your code is not a great way of timing code. Timing code is
>> *very* difficult, and can be effected by many things, such as external
>> processes, CPU caches, even the function you use for getting the time.
>> Much of the time you are timing here will be in creating the range(loops)
>> list, especially if loops is big.
> 
> But both loops are the same size. And that overhead can quickly be
> disposed of by measuring empty loops in both cases. (On my machine,
> about 0.006/7 seconds for loops of 100,000.)

No, you cannot make that assumption, not in general. On modern machines, you
cannot assume that the time it takes to execute foo() immediately followed
by bar() is the same as the time it takes to execute foo() and bar()
separately.

Modern machines run multi-tasking operating systems, where there can be
other processes running. Depending on what you use as your timer, you may
be measuring the time that those other processes run. The OS can cache
frequently used pieces of code, which allows it to run faster. The CPU
itself will cache some code.

The shorter the code snippet, the more these complications are relevant. In
this particular case, we can be reasonably sure that the time it takes to
create a list range(10000) and the overhead of the loop is *probably* quite
a small percentage of the time it takes to perform 100000 vector
multiplications. But that's not a safe assumption for all code snippets.

This is why the timeit module exists: to do the right thing when it matters,
so that you don't have to think about whether or not it matters. The timeit
module works really really hard to get good quality, accurate timings,
minimizing any potential overhead.

The timeit module automates a bunch of tricky-to-right best practices for
timing code. Is that a problem?



>> The best way to time small snippets of code is to use the timeit module.
>> Open a terminal or shell (*not* the Python interactive interpreter, the
>> operating system's shell: you should expect a $ or % prompt) and run
>> timeit from that. Copy and paste the following two commands into your
>> shell prompt:
>>
>>
>> python2.7 -m timeit --repeat 5 -s "import numpy as np" \
>> -s "x = np.array([1, 2, 3])" -s "y = np.array([4, 5, 6])" \
>> -- "np.cross(x, y)"
[...]

> Yes, I can see that typing all the code out again, and remembering all
> those options and putting -s, -- and \ in all the right places, is a
> much better way of doing it! Not error prone at all.

Gosh Bart, how did you manage to write that sentence? How did you remember
all those words, and remember to put the punctuation marks in the right
places?

You even used sarcasm! You must be a genius. (Oh look, I can use it too.)

Seriously Bart? You've been a programmer for how many decades, and you can't
work out how to call a command from the shell? This is about working
effectively with your tools, and a basic understanding of the shell is an
essential tool for programmers.

This was a *simple* command. It was a LONG command, but don't be fooled by
the length, and the fact that it went over multiple lines, it was dirt
simple. I'm not saying that every programmer needs to be a greybeard Unix
guru (heaven knows that I'm not!), but they ought to be able to run simple
commands from the command line.

Those who don't are in the same position as carpenters who don't know the
differences between the various kinds of hammer or saws. Sure, you can
still do a lot of work using just one kind of hammer and one kind of saw,
but you'll work better, faster and smarter with the right kind. You don't
use a rip saw to make fine cuts, and you don't use a mash hammer to drive
tacks.

The -m option lets you run a module without knowing the precise location of
the source file. Some of the most useful commands to learn:

python -m unittest ...
python -m doctest ...
python -m timeit ...

The biggest advantage of calling the timeit module is that it will
automatically select the number of iterations you need to run to get good
timing results, without wasting time running excessive loops.

(The timeit module is *significantly* improved in Python 3.6, but even in
older versions its pretty good.)

But if you prefer doing it "old school" from within Python, then:

from timeit import Timer
t = Timer('np.cross(x, y)',  setup="""
import numpy as np
x = np.array([1, 2, 3])
y = np.array([4, 5, 6])
""")

# take five measurements of 100000 calls each, and report the fastest
result = min(t.repeat(number=100000, repeat=5))/100000
print(result)  # time in seconds per call



Better?




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From bc at freeuk.com  Mon Nov 21 09:53:35 2016
From: bc at freeuk.com (BartC)
Date: Mon, 21 Nov 2016 14:53:35 +0000
Subject: Numpy slow at vector cross product?
In-Reply-To: <mailman.18.1479732277.3146.python-list@python.org>
References: <o0t22o$d9r$3@dont-email.me>
 <58326062$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <o0uq77$cjf$1@blaine.gmane.org>
 <mailman.18.1479732277.3146.python-list@python.org>
Message-ID: <o0v1oe$jju$1@dont-email.me>

On 21/11/2016 12:44, Peter Otten wrote:

> After a look into the source this is no longer a big surprise (numpy 1.8.2):
>
>     if axis is not None:
>         axisa, axisb, axisc=(axis,)*3
>     a = asarray(a).swapaxes(axisa, 0)
>     b = asarray(b).swapaxes(axisb, 0)

>
> The situation may be different when you process vectors in bulk, i. e.
> instead of [cross(a, bb) for bb in b] just say cross(a, b).

I thought that numpy was supposed to be fast? Also that the critical 
bits were not implemented in Python?

Anyway I tried your code (put into a function as shown below) in the 
same test program, and it was *still* 3 times as fast as numpy! 
(mycross() was 3 times as fast as np.cross().)

Explain that...


---------------------------------------

def mycross(a,b,axisa=-1, axisb=-1, axisc=-1, axis=None):
     if axis is not None:
         axisa, axisb, axisc=(axis,)*3
     a = np.asarray(a).swapaxes(axisa, 0)
     b = np.asarray(b).swapaxes(axisb, 0)
     msg = "incompatible dimensions for cross product\n"\
           "(dimension must be 2 or 3)"
     if (a.shape[0] not in [2, 3]) or (b.shape[0] not in [2, 3]):
         raise ValueError(msg)
     if a.shape[0] == 2:
         if (b.shape[0] == 2):
             cp = a[0]*b[1] - a[1]*b[0]
             if cp.ndim == 0:
                 return cp
             else:
                 return cp.swapaxes(0, axisc)
         else:
             x = a[1]*b[2]
             y = -a[0]*b[2]
             z = a[0]*b[1] - a[1]*b[0]
     elif a.shape[0] == 3:
         if (b.shape[0] == 3):
             x = a[1]*b[2] - a[2]*b[1]
             y = a[2]*b[0] - a[0]*b[2]
             z = a[0]*b[1] - a[1]*b[0]
         else:
             x = -a[2]*b[1]
             y = a[2]*b[0]
             z = a[0]*b[1] - a[1]*b[0]
     cp = np.array([x, y, z])
     if cp.ndim == 1:
         return cp
     else:
         return cp.swapaxes(0, axisc)

---------------------------------------
Tested as:

	x=np.array([1,2,3])
	y=np.array([4,5,6])

	start=time.clock()
	for i in range(loops):
	    z=mycross(x,y)
	print "Calc, %s loops: %.2g seconds" %(loops,time.clock()-start)


-- 
Bartc


From vek.m1234 at gmail.com  Mon Nov 21 10:16:15 2016
From: vek.m1234 at gmail.com (Veek M)
Date: Mon, 21 Nov 2016 20:46:15 +0530
Subject: What is the difference between class Foo(): and class
 Date(object):
References: <o0uoff$ht5$1@dont-email.me>
 <58330109$0$1609$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o0v32t$nog$1@dont-email.me>

Steve D'Aprano wrote:

> On Mon, 21 Nov 2016 11:15 pm, Veek M wrote:
> 
>>>>> class Foo():
>> ...  pass
>> ...
>>>>> class Bar(Foo):
>> ...  pass
>> ...
>>>>> b = Bar()
>>>>> type(b)
>> <type 'instance'>
> [...]
> 
>> What is going on here? Shouldn't x = EuroDate();  type(x) give
>> 'instance'?? Why is 'b' an 'instance' and 'x' EuroDate?
>> Why isn't 'b' Bar?
> 
> 
> It looks like you are running Python 2, and have stumbled across an
> annoyance from the earliest days of Python: the "classic", or
> "old-style", class.
> 
> Before Python 2.2, custom classes and built-in types like int, float,
> dict and list were different. You have just discovered one of the ways
> they were different: instances of custom classes all had the same
> type, even if the class was different:
> 
> # Python 2
> 
> py> class Dog:
> ...     pass
> ...
> py> class Cat:
> ...     pass
> ...
> py> lassie = Dog()
> py> garfield = Cat()
> py> type(lassie) is type(garfield)
> True
> py> type(lassie)
> <type 'instance'>
> 
> 
> This is just the most obvious difference between "classic classes" and
> types. Some of the other differences:
> 
> - The method resolution order (MRO) is different: the classic class
>   MRO is buggy for diamond-shaped multiple inheritance, and special
>   dunder methods like __eq__ are resolved slightly differently.
> 
> - super, properties, class methods and static methods don't work for
>   classic classes.
> 
> - The metaclass of classic classes is different:
> 
> py> type(Dog)
> <type 'classobj'>
> py> type(float)
> <type 'type'>
> 
> - Attribute lookup for classic classes is slightly different; in
>   particular, the special __getattribute__ method doesn't work.
> 
> 
> In Python 2.2, the built-in types (list, dict, float etc) were unified
> with the class mechanism, but for backwards compatibility the
> old-style classes had to be left in. So Python had two class
> mechanisms:
> 
> - "New-style classes", or types, inherit from object, or some
>   other built-in type, and support properties, etc.
> 
> - "Old-style classes", don't inherit from object, don't support
>   properties etc.
> 
> 
> So in Python 2, when you write:
> 
>     class Foo:
> 
> or
> 
>     class Foo():
> 
> 
> you get an old-style class. But when you inherit from object, you get
> a new-style class. Classic classes are an obsolete feature from Python
> 2. They are removed in Python 3, and things are much simpler. In
> Python 3, it doesn't matter whether you write:
> 
>     class Foo:
>     class Foo():
>     class Foo(object):
> 
> the result is the same: a new-style class, or type.
> 
> The best thing to do in Python 2 is to always, without exception,
> write
> 
>     class Foo(object):
> 
> to define your base classes. That will ensure that property, super,
> classmethod, staticmethod, __getattribute__, etc. will all work
> correctly, and you will avoid the surprises of classic classes.
> 
> 
> 
Thanks guys, got it!


From skip.montanaro at gmail.com  Mon Nov 21 10:34:52 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Mon, 21 Nov 2016 09:34:52 -0600
Subject: Numpy slow at vector cross product?
In-Reply-To: <o0v1oe$jju$1@dont-email.me>
References: <o0t22o$d9r$3@dont-email.me>
 <58326062$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <o0uq77$cjf$1@blaine.gmane.org>
 <mailman.18.1479732277.3146.python-list@python.org>
 <o0v1oe$jju$1@dont-email.me>
Message-ID: <CANc-5UwxfiCsbUGo7F+pqOtf+43b+ef=6xiKVa2b5tRkTBf6rw@mail.gmail.com>

Perhaps your implementation isn't as general as numpy's? I pulled out
the TestCross class from numpy.core.tests.test_numeric and replaced
calls to np.cross with calls to your function. I got an error in
test_broadcasting_shapes:

ValueError: operands could not be broadcast together with shapes (1,2) (5,)

Skip


From nobody at nowhere.invalid  Mon Nov 21 12:04:32 2016
From: nobody at nowhere.invalid (Nobody)
Date: Mon, 21 Nov 2016 17:04:32 +0000
Subject: Numpy slow at vector cross product?
References: <o0t22o$d9r$3@dont-email.me>
 <58326062$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <o0uq77$cjf$1@blaine.gmane.org>
 <mailman.18.1479732277.3146.python-list@python.org>
 <o0v1oe$jju$1@dont-email.me>
Message-ID: <pan.2016.11.21.17.04.32.390000@nowhere.invalid>

On Mon, 21 Nov 2016 14:53:35 +0000, BartC wrote:

> Also that the critical bits were not implemented in Python?

That is correct. You'll notice that there aren't any loops in numpy.cross.
It's just a wrapper around a bunch of vectorised operations (*, -, []).

If you aren't taking advantage of vectorisation, there's no reason to
expect numpy to be any faster than primitive operations, any more than
you'd expect

	(numpy.array([1]) + numpy.array([2])[0]

to be faster than "1+2".

Beyond that, you'd expect a generic function to be at a disadvantage
compared to a function which makes assumptions about its arguments.
Given what it does, I wouldn't expect numpy.cross() to be faster for
individual vectors if it was written in C.



From porton at narod.ru  Mon Nov 21 12:48:10 2016
From: porton at narod.ru (Victor Porton)
Date: Mon, 21 Nov 2016 19:48:10 +0200
Subject: Which of two variants of code is better?
Message-ID: <o0vc0p$nul$1@gioia.aioe.org>

Which of two variants of code to construct an "issue comment" object (about 
BitBucket issue comments) is better?

1.

obj = IssueComment(Issue(IssueGroup(repository, 'issues'), id1), id2)

or

2.

list = [('issues', IssueGroup), (id1, Issue), (id2, IssueComment)]
obj = construct_subobject(repository, list)

(`construct_subobject` is to be defined in such as way that "1" and "2" do 
the same.)

Would you advise me to make such function construct_subobject function or 
just to use the direct coding as in "1"?

-- 
Victor Porton - http://portonvictor.org


From ned at nedbatchelder.com  Mon Nov 21 13:26:47 2016
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Mon, 21 Nov 2016 10:26:47 -0800 (PST)
Subject: Which of two variants of code is better?
In-Reply-To: <o0vc0p$nul$1@gioia.aioe.org>
References: <o0vc0p$nul$1@gioia.aioe.org>
Message-ID: <ca88fb83-23f5-45a4-8c5d-ee31ee207d34@googlegroups.com>

On Monday, November 21, 2016 at 12:48:25 PM UTC-5, Victor Porton wrote:
> Which of two variants of code to construct an "issue comment" object (about 
> BitBucket issue comments) is better?
> 
> 1.
> 
> obj = IssueComment(Issue(IssueGroup(repository, 'issues'), id1), id2)
> 
> or
> 
> 2.
> 
> list = [('issues', IssueGroup), (id1, Issue), (id2, IssueComment)]
> obj = construct_subobject(repository, list)
> 
> (`construct_subobject` is to be defined in such as way that "1" and "2" do 
> the same.)
> 
> Would you advise me to make such function construct_subobject function or 
> just to use the direct coding as in "1"?

Neither of these seem very convenient. I don't know what an IssueGroup is,
so I don't know why I need to specify it.  To create a comment on an issue,
why do I need id2, which seems to be the id of a comment?

How about this:

    obj = IssueComment(repo=repository, issue=id1)

or:

    obj = repository.create_issue_comment(issue=id1)

--Ned.


From bc at freeuk.com  Mon Nov 21 13:43:49 2016
From: bc at freeuk.com (BartC)
Date: Mon, 21 Nov 2016 18:43:49 +0000
Subject: Numpy slow at vector cross product?
In-Reply-To: <pan.2016.11.21.17.04.32.390000@nowhere.invalid>
References: <o0t22o$d9r$3@dont-email.me>
 <58326062$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <o0uq77$cjf$1@blaine.gmane.org>
 <mailman.18.1479732277.3146.python-list@python.org>
 <o0v1oe$jju$1@dont-email.me> <pan.2016.11.21.17.04.32.390000@nowhere.invalid>
Message-ID: <o0vf84$ble$1@dont-email.me>

On 21/11/2016 17:04, Nobody wrote:
> On Mon, 21 Nov 2016 14:53:35 +0000, BartC wrote:
>
>> Also that the critical bits were not implemented in Python?
>
> That is correct. You'll notice that there aren't any loops in numpy.cross.
> It's just a wrapper around a bunch of vectorised operations (*, -, []).
>
> If you aren't taking advantage of vectorisation, there's no reason to
> expect numpy to be any faster than primitive operations, any more than
> you'd expect
>
> 	(numpy.array([1]) + numpy.array([2])[0]
>
> to be faster than "1+2".
>
> Beyond that, you'd expect a generic function to be at a disadvantage
> compared to a function which makes assumptions about its arguments.
> Given what it does, I wouldn't expect numpy.cross() to be faster for
> individual vectors if it was written in C.

The fastest I can get compiled, native code to do this is at 250 million 
cross-products per second.

The fastest using pure Python executed with Python 2.7 is 0.5 million 
per second.

With pypy, around 8 million per second. (Results will vary by machine, 
version, and OS so this is just one set of timings.)

So numpy, at 0.03 million per second [on a different OS and different 
version], has room for improvement I think!

(In all cases, the loop has been hobbled so that one component 
increments per loop, and one component of the result is summed and then 
displayed at the end.

This is to stop gcc, and partly pypy, from optimising the code out of 
existence; usually you are not calculating the same vector product 
repeatedly. Without the restraint, pypy leaps to 100 million per second, 
and gcc to an infinite number.)

The tests were with values assumed to be vectors, assumed to have 3 
components, and without any messing about with axes, whatever that code 
does. It's just a pure, streamlined, vector cross product (just as I use 
in my own code).

Such a streamlined version can also be written in Python. (Although it 
would be better with a dedicated 3-component vector type rather than a 
general purpose list or even numpy array.)

It's still a puzzle why directly executing the code that numpy uses was 
still faster than numpy itself, when both were run with CPython. Unless 
numpy is perhaps using extra wrappers around numpy.cross.

-- 
Bartc


From porton at narod.ru  Mon Nov 21 13:46:46 2016
From: porton at narod.ru (Victor Porton)
Date: Mon, 21 Nov 2016 20:46:46 +0200
Subject: Which of two variants of code is better?
References: <o0vc0p$nul$1@gioia.aioe.org>
 <ca88fb83-23f5-45a4-8c5d-ee31ee207d34@googlegroups.com>
Message-ID: <o0vfek$u1m$1@gioia.aioe.org>

Ned Batchelder wrote:

> On Monday, November 21, 2016 at 12:48:25 PM UTC-5, Victor Porton wrote:
>> Which of two variants of code to construct an "issue comment" object
>> (about BitBucket issue comments) is better?
>> 
>> 1.
>> 
>> obj = IssueComment(Issue(IssueGroup(repository, 'issues'), id1), id2)
>> 
>> or
>> 
>> 2.
>> 
>> list = [('issues', IssueGroup), (id1, Issue), (id2, IssueComment)]
>> obj = construct_subobject(repository, list)
>> 
>> (`construct_subobject` is to be defined in such as way that "1" and "2"
>> do the same.)
>> 
>> Would you advise me to make such function construct_subobject function or
>> just to use the direct coding as in "1"?
> 
> Neither of these seem very convenient. I don't know what an IssueGroup is,

It is a helper object which helps to paginate issues.

> so I don't know why I need to specify it.  To create a comment on an
> issue, why do I need id2, which seems to be the id of a comment?

It does not create a comment. It is preparing to load the comment.

> How about this:
> 
>     obj = IssueComment(repo=repository, issue=id1)
> 
> or:
> 
>     obj = repository.create_issue_comment(issue=id1)

Your code is too specialized. I want to make all my code following the same 
patterns. (And I am not going to define helper methods like yours, because 
we do not use it often enough to be worth of a specific method.)

-- 
Victor Porton - http://portonvictor.org


From subhabangalore at gmail.com  Mon Nov 21 14:27:31 2016
From: subhabangalore at gmail.com (subhabangalore at gmail.com)
Date: Mon, 21 Nov 2016 11:27:31 -0800 (PST)
Subject: Working around multiple files in a folder
Message-ID: <a8b89adf-c529-4f01-ad4b-362fb087a489@googlegroups.com>

I have a python script where I am trying to read from a list of files in a folder and trying to process something. 
As I try to take out the output I am presently appending to a list.

But I am trying to write the result of individual files in individual list or files.

The script is as follows:

import glob
def speed_try():
    #OPENING THE DICTIONARY
    a4=open("/python27/Dictionaryfile","r").read()
    #CONVERTING DICTIONARY INTO WORDS
    a5=a4.lower().split()
    list1=[]
    for filename in glob.glob('/Python27/*.txt'):
        a1=open(filename,"r").read()
        a2=a1.lower()
        a3=a2.split()
        for word in a3:
            if word in a5:
                a6=a5.index(word)
                a7=a6+1
                a8=a5[a7]
                a9=word+"/"+a8
                list1.append(a9)
            elif word not in a5:
                list1.append(word)
            else:
                print "None"

    x1=list1
    x2=" ".join(x1)
    print x2

Till now, I have tried to experiment over the following solutions:

a) def speed_try():
      #OPENING THE DICTIONARY
      a4=open("/python27/Dictionaryfile","r").read()
      #CONVERTING DICTIONARY INTO WORDS
      a5=a4.lower().split()
      list1=[]
      for filename in glob.glob('/Python27/*.txt'):
         a1=open(filename,"r").read()
         a2=a1.lower()
         a3=a2.split()
          list1.append(a3)
        

    x1=list1
    print x1

Looks very close but I am unable to fit the if...elif...else part. 

b) import glob
def multi_filehandle():
    list_of_files = glob.glob('/Python27/*.txt')
    for file_name in list_of_files:
        FI = open(file_name, 'r')
        FI1=FI.read().split()
        FO = open(file_name.replace('txt', 'out'), 'w') 
        for line in FI:
            FO.write(line)

        FI.close()
        FO.close()

I could write output but failing to do processing of the files between opening and writing.

I am trying to get examples from fileinput.

If anyone of the learned members may kindly suggest how may I proceed.

I am using Python2.x on MS-Windows. 

The practices are scripts and not formal codes so I have not followed style guides.

Apology for any indentation error.

Thanking in advance.




From emilevansebille at gmail.com  Mon Nov 21 16:19:00 2016
From: emilevansebille at gmail.com (Emile van Sebille)
Date: Mon, 21 Nov 2016 13:19:00 -0800
Subject: Working around multiple files in a folder
In-Reply-To: <a8b89adf-c529-4f01-ad4b-362fb087a489@googlegroups.com>
References: <a8b89adf-c529-4f01-ad4b-362fb087a489@googlegroups.com>
Message-ID: <o0voc0$qg1$1@blaine.gmane.org>

On 11/21/2016 11:27 AM, subhabangalore at gmail.com wrote:
> I have a python script where I am trying to read from a list of files in a folder and trying to process something.
> As I try to take out the output I am presently appending to a list.
>
> But I am trying to write the result of individual files in individual list or files.
>
> The script is as follows:
>
> import glob
> def speed_try():
>      #OPENING THE DICTIONARY
>      a4=open("/python27/Dictionaryfile","r").read()
>      #CONVERTING DICTIONARY INTO WORDS
>      a5=a4.lower().split()
>      list1=[]
>      for filename in glob.glob('/Python27/*.txt'):
>          a1=open(filename,"r").read()
>          a2=a1.lower()
>          a3=a2.split()
>          for word in a3:
>              if word in a5:
>                  a6=a5.index(word)
>                  a7=a6+1
>                  a8=a5[a7]
>                  a9=word+"/"+a8
>                  list1.append(a9)
>              elif word not in a5:
>                  list1.append(word)
>              else:
>                  print "None"
>
>      x1=list1
>      x2=" ".join(x1)
>      print x2
>
> Till now, I have tried to experiment over the following solutions:
>
> a) def speed_try():
>        #OPENING THE DICTIONARY
>        a4=open("/python27/Dictionaryfile","r").read()
>        #CONVERTING DICTIONARY INTO WORDS
>        a5=a4.lower().split()
>        list1=[]
>        for filename in glob.glob('/Python27/*.txt'):
>           a1=open(filename,"r").read()
>           a2=a1.lower()
>           a3=a2.split()
>            list1.append(a3)
>
>
>      x1=list1
>      print x1
>
> Looks very close but I am unable to fit the if...elif...else part.
>
> b) import glob
> def multi_filehandle():
>      list_of_files = glob.glob('/Python27/*.txt')
>      for file_name in list_of_files:
>          FI = open(file_name, 'r')
>          FI1=FI.read().split()
>          FO = open(file_name.replace('txt', 'out'), 'w')
>          for line in FI:

at this point, there's nothing left to be read from FI having been fully 
drained to populate FI1 -- maybe you want to loop over FI1 instead?

Emile


>              FO.write(line)
>
>          FI.close()
>          FO.close()
>
> I could write output but failing to do processing of the files between opening and writing.
>
> I am trying to get examples from fileinput.
>
> If anyone of the learned members may kindly suggest how may I proceed.
>
> I am using Python2.x on MS-Windows.
>
> The practices are scripts and not formal codes so I have not followed style guides.
>
> Apology for any indentation error.
>
> Thanking in advance.
>
>




From blindanagram at nowhere.com  Mon Nov 21 17:45:36 2016
From: blindanagram at nowhere.com (BlindAnagram)
Date: Mon, 21 Nov 2016 22:45:36 +0000
Subject: Enigma 1140 - link problem
Message-ID: <KcudnUvsUaWN5K7FnZ2dnUU78VPNnZ2d@brightview.co.uk>

Hi Jim,

In my comment/solution for this Enigma I tried to post a link to my
number theory library but my HTML got removed.

Could you please replace the first sentence with:

A solution using my <a
href="http://173.254.28.24/~brgladma/number_theory.py">number theory</a>
library:

(without the line wrapping).

  thanks

     Brian


From fillmore_remove at hotmail.com  Mon Nov 21 18:27:51 2016
From: fillmore_remove at hotmail.com (Fillmore)
Date: Mon, 21 Nov 2016 18:27:51 -0500
Subject: MemoryError and Pickle
Message-ID: <o0vvtm$1rpo$1@gioia.aioe.org>


Hi there, Python newbie here.

I am working with large files. For this reason I figured that I would 
capture the large input into a list and serialize it with pickle for 
later (faster) usage.
Everything has worked beautifully until today when the large data (1GB) 
file caused a MemoryError :(

Question for experts: is there a way to refactor this so that data may 
be filled/written/released as the scripts go and avoid the problem?
code below.

Thanks

data = list()
for line in sys.stdin:

     try:
         parts = line.strip().split("\t")
         t = parts[0]
         w = parts[1]
         u = parts[2]



         #let's retain in-memory copy of data
         data.append({"ta": t,
                      "wa": w,
                      "ua": u
         })

     except IndexError:
         print("Problem with line :"+line, file=sys.stderr)
         pass

#time to save data object into a pickle file

fileObject = open(filename,"wb")
pickle.dump(data,fileObject)
fileObject.close()


From gordon at panix.com  Mon Nov 21 18:43:32 2016
From: gordon at panix.com (John Gordon)
Date: Mon, 21 Nov 2016 23:43:32 +0000 (UTC)
Subject: MemoryError and Pickle
References: <o0vvtm$1rpo$1@gioia.aioe.org>
Message-ID: <o100r4$qmh$1@reader2.panix.com>

In <o0vvtm$1rpo$1 at gioia.aioe.org> Fillmore <fillmore_remove at hotmail.com> writes:


> Question for experts: is there a way to refactor this so that data may 
> be filled/written/released as the scripts go and avoid the problem?
> code below.

That depends on how the data will be read.  Here is one way to do it:

    fileObject = open(filename, "w")
    for line in sys.stdin:
        parts = line.strip().split("\t")
        fileObject.write("ta: %s\n" % parts[0])
        fileObject.write("wa: %s\n" % parts[1])
        fileObject.write("ua: %s\n" % parts[2])
    fileObject.close()

But this doesn't use pickle format, so your reader program would have to
be modified to read this format.  And you'll run into the same problem if
the reader expects to keep all the data in memory.

-- 
John Gordon                   A is for Amy, who fell down the stairs
gordon at panix.com              B is for Basil, assaulted by bears
                                -- Edward Gorey, "The Gashlycrumb Tinies"



From __peter__ at web.de  Mon Nov 21 19:40:26 2016
From: __peter__ at web.de (Peter Otten)
Date: Tue, 22 Nov 2016 01:40:26 +0100
Subject: MemoryError and Pickle
References: <o0vvtm$1rpo$1@gioia.aioe.org>
Message-ID: <o1045n$k52$1@blaine.gmane.org>

Fillmore wrote:

> Hi there, Python newbie here.
> 
> I am working with large files. For this reason I figured that I would
> capture the large input into a list and serialize it with pickle for
> later (faster) usage.

But is it really faster? If the pickle is, let's say, twice as large as the 
original file it should take roughly twice as long to read the data...




From ckaynor at zindagigames.com  Mon Nov 21 19:49:19 2016
From: ckaynor at zindagigames.com (Chris Kaynor)
Date: Mon, 21 Nov 2016 16:49:19 -0800
Subject: MemoryError and Pickle
In-Reply-To: <o100r4$qmh$1@reader2.panix.com>
References: <o0vvtm$1rpo$1@gioia.aioe.org> <o100r4$qmh$1@reader2.panix.com>
Message-ID: <CALvWhxsc0FCJo8z6KA_khtAkHxRtmfgWUAQ4FgKu3pUZkPYYoQ@mail.gmail.com>

On Mon, Nov 21, 2016 at 3:43 PM, John Gordon <gordon at panix.com> wrote:
> In <o0vvtm$1rpo$1 at gioia.aioe.org> Fillmore <fillmore_remove at hotmail.com> writes:
>
>
>> Question for experts: is there a way to refactor this so that data may
>> be filled/written/released as the scripts go and avoid the problem?
>> code below.
>
> That depends on how the data will be read.  Here is one way to do it:
>
>     fileObject = open(filename, "w")
>     for line in sys.stdin:
>         parts = line.strip().split("\t")
>         fileObject.write("ta: %s\n" % parts[0])
>         fileObject.write("wa: %s\n" % parts[1])
>         fileObject.write("ua: %s\n" % parts[2])
>     fileObject.close()
>
> But this doesn't use pickle format, so your reader program would have to
> be modified to read this format.  And you'll run into the same problem if
> the reader expects to keep all the data in memory.

If you want to keep using pickle, you should be able to pickle each
item of the list to the file one at a time. As long as the file is
kept open (or seeked to the end), you should be able to dump without
overwriting the old data, and read starting at the end of the previous
pickle stream.

I haven't tested it, so there may be issues (if it fails, you can try
using dumps and writing to the file by hand):

Writing:
with open(filename, 'wb') as fileObject:
    for line in sys.stdin:
        pickle.dump(line, fileObject)

Reading:
with open(filename, 'wb') as fileObject:
    while not fileObject.eof: # Not sure of the correct syntax, but
gives the idea
        line = pickle.load(fileObject)
        # do something with line


It should also be noted that if you do not need to support multiple
Python versions, you may want to specify a protocol to pickle.dump to
use a better version of the format. -1 will use the latest (best if
you only care about one version of Python.); 4 is currently the latest
version (added in 3.4), which may be useful if you need
forward-compatibility but not backwards-compatibility. 2 is the latest
version available in Python 2 (added in Python 2.3) See
https://docs.python.org/3.6/library/pickle.html#data-stream-format for
more information.


From steve+python at pearwood.info  Mon Nov 21 19:59:50 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Tue, 22 Nov 2016 11:59:50 +1100
Subject: MemoryError and Pickle
References: <o0vvtm$1rpo$1@gioia.aioe.org> <o1045n$k52$1@blaine.gmane.org>
 <mailman.26.1479775234.3146.python-list@python.org>
Message-ID: <58339888$0$1611$c3e8da3$5496439d@news.astraweb.com>

On Tue, 22 Nov 2016 11:40 am, Peter Otten wrote:

> Fillmore wrote:
> 
>> Hi there, Python newbie here.
>> 
>> I am working with large files. For this reason I figured that I would
>> capture the large input into a list and serialize it with pickle for
>> later (faster) usage.
> 
> But is it really faster? If the pickle is, let's say, twice as large as
> the original file it should take roughly twice as long to read the data...

But the code is more complex, therefore faster.

That's how it works, right?

*wink*




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From tjreedy at udel.edu  Mon Nov 21 20:18:39 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 21 Nov 2016 20:18:39 -0500
Subject: Enigma 1140 - link problem
In-Reply-To: <KcudnUvsUaWN5K7FnZ2dnUU78VPNnZ2d@brightview.co.uk>
References: <KcudnUvsUaWN5K7FnZ2dnUU78VPNnZ2d@brightview.co.uk>
Message-ID: <b738731c-e4c5-6ab8-1f6f-5ab75bf13426@udel.edu>

On 11/21/2016 5:45 PM, BlindAnagram wrote:
> Hi Jim,
...
>   thanks
>      Brian

'Brian', you sent this to python-list instead of Jim.  If this is not 
spam, try again with a different 'To:'


-- 
Terry Jan Reedy



From steve+python at pearwood.info  Mon Nov 21 20:40:19 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Tue, 22 Nov 2016 12:40:19 +1100
Subject: MemoryError and Pickle
References: <o0vvtm$1rpo$1@gioia.aioe.org>
Message-ID: <5833a206$0$1610$c3e8da3$5496439d@news.astraweb.com>

On Tue, 22 Nov 2016 10:27 am, Fillmore wrote:

> 
> Hi there, Python newbie here.
> 
> I am working with large files. For this reason I figured that I would
> capture the large input into a list and serialize it with pickle for
> later (faster) usage.
> Everything has worked beautifully until today when the large data (1GB)
> file caused a MemoryError :(

At what point do you run out of memory? When building the list? If so, then
you need more memory, or smaller lists, or avoid creating a giant list in
the first place.

If you can successfully build the list, but then run out of memory when
trying to pickle it, then you may need another approach.

But as always, to really be sure what is going on, we need to see the full
traceback (not just the "MemoryError" part) and preferably a short, simple
example that replicates the error:

http://www.sscce.org/



> Question for experts: is there a way to refactor this so that data may
> be filled/written/released as the scripts go and avoid the problem?

I'm not sure what you are doing with this data. I guess you're not just:

- read the input, one line at a time
- create a giant data list
- pickle the list

and then never look at the pickle again.

I imagine that you want to process the list in some way, but how and where
and when is a mystery. But most likely you will later do:

- unpickle the list, creating a giant data list again
- process the data list

So I'm not sure what advantage the pickle is, except as make-work. Maybe
I've missed something, but if you're running out of memory processing the
giant list, perhaps a better approach is:

- read the input, one line at a time
- process that line


and avoid building the giant list or the pickle at all.


> code below.
> 
> Thanks
>
> data = list()
> for line in sys.stdin:
>      try:
>          parts = line.strip().split("\t")
>          t = parts[0]
>          w = parts[1]
>          u = parts[2]
>          #let's retain in-memory copy of data
>          data.append({"ta": t,
>                       "wa": w,
>                       "ua": u
>          })
>      except IndexError:
>          print("Problem with line :"+line, file=sys.stderr)
>          pass
> 
> #time to save data object into a pickle file
> 
> fileObject = open(filename,"wb")
> pickle.dump(data,fileObject)
> fileObject.close()

Let's re-write some of your code to make it better:

data = []
for line in sys.stdin:
    try:
        t, w, u = line.strip().split("\t")
    except ValueError as err:
        print("Problem with line:", line, file=sys.stderr)
    data.append({"ta": t, "wa": w, "ua": u})

with open(filename, "wb") as fileObject:
    pickle.dump(data, fileObject)


Its not obvious where you are running out of memory, but my guess is that it
is most likely while building the giant list. You have a LOT of small
dicts, each one with exactly the same set of keys. You can probably save a
lot of memory by using a tuple, or better, a namedtuple.

py> from collections import namedtuple
py> struct = namedtuple("struct", "ta wa ua")
py> x = struct("abc", "def", "ghi")
py> y = {"ta": "abc", "wa": "def", "ua": "ghi"}
py> sys.getsizeof(x)
36
py> sys.getsizeof(y)
144


So each of those little dicts {"ta": t, "wa": w, "ua": u} in your list
potentially use as much as four times the memory as a namedtuple would use.
So using namedtuple might very well save enough memory to avoid the
MemoryError altogether.


from collections import namedtuple
struct = namedtuple("struct", "ta wa ua")
data = []
for line in sys.stdin:
    try:
        t, w, u = line.strip().split("\t")
    except ValueError as err:
        print("Problem with line:", line, file=sys.stderr)
    data.append(struct(t, w, a))

with open(filename, "wb") as fileObject:
    pickle.dump(data, fileObject)


And as a bonus, when you come to use the record, instead of having to write:

    line["ta"]

to access the first field, you can write:

    line.ta



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From bc at freeuk.com  Mon Nov 21 20:45:50 2016
From: bc at freeuk.com (BartC)
Date: Tue, 22 Nov 2016 01:45:50 +0000
Subject: Numpy slow at vector cross product?
In-Reply-To: <583309a9$0$1596$c3e8da3$5496439d@news.astraweb.com>
References: <o0t22o$d9r$3@dont-email.me>
 <58326062$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <o0uo4r$gno$1@dont-email.me>
 <583309a9$0$1596$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o107ve$3ad$1@dont-email.me>

On 21/11/2016 14:50, Steve D'Aprano wrote:
> On Mon, 21 Nov 2016 11:09 pm, BartC wrote:

> Modern machines run multi-tasking operating systems, where there can be
> other processes running. Depending on what you use as your timer, you may
> be measuring the time that those other processes run. The OS can cache
> frequently used pieces of code, which allows it to run faster. The CPU
> itself will cache some code.

You get to know after while what kinds of processes affect timings. For 
example, streaming a movie at the same time. So when you need to compare 
timings, you turn those off.

> The shorter the code snippet, the more these complications are relevant. In
> this particular case, we can be reasonably sure that the time it takes to
> create a list range(10000) and the overhead of the loop is *probably* quite
> a small percentage of the time it takes to perform 100000 vector
> multiplications. But that's not a safe assumption for all code snippets.

Yes, it was one of those crazy things that Python used to have to do, 
creating a list of N numbers just in order to be able to count to N.

But that's not significant here. Either experience, or a preliminary 
test with an empty loop, or using xrange, or using Py3, will show that 
the loop overheads for N iterations in this case are small in comparison 
to executing the bodies of the loops.

> This is why the timeit module exists: to do the right thing when it matters,
> so that you don't have to think about whether or not it matters. The timeit
> module works really really hard to get good quality, accurate timings,
> minimizing any potential overhead.
>
> The timeit module automates a bunch of tricky-to-right best practices for
> timing code. Is that a problem?

The problem is it substitutes a bunch of tricky-to-get-right options and 
syntax which has to to typed /at the command line/. And you really don't 
want to have to write code at the command line (especially if sourced 
from elsewhere, which means you have to transcribe it).


> But if you prefer doing it "old school" from within Python, then:
>
> from timeit import Timer
> t = Timer('np.cross(x, y)',  setup="""
> import numpy as np
> x = np.array([1, 2, 3])
> y = np.array([4, 5, 6])
> """)
>
> # take five measurements of 100000 calls each, and report the fastest
> result = min(t.repeat(number=100000, repeat=5))/100000
> print(result)  # time in seconds per call

> Better?

A bit, but the code is now inside a string!

Code will normally exist as a proper part of a module, not on the 
command line, in a command history, or in a string, so why not test it 
running inside a module?

But I've done a lot of benchmarking and actually measuring execution 
time is just part of it. This test I ran from inside a function for 
example, not at module-level, as that is more typical.

Are the variables inside a time-it string globals or locals? It's just a 
lot of extra factors to worry about, and extra things to get wrong.

The loop timings used by the OP showed one took considerably longer than 
the other. And that was confirmed by others. There's nothing wrong with 
that method.

-- 
Bartc



From steve+python at pearwood.info  Mon Nov 21 21:44:48 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Tue, 22 Nov 2016 13:44:48 +1100
Subject: Numpy slow at vector cross product?
References: <o0t22o$d9r$3@dont-email.me>
 <58326062$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <o0uq77$cjf$1@blaine.gmane.org>
 <mailman.18.1479732277.3146.python-list@python.org>
 <o0v1oe$jju$1@dont-email.me> <pan.2016.11.21.17.04.32.390000@nowhere.invalid>
 <o0vf84$ble$1@dont-email.me>
Message-ID: <5833b121$0$1603$c3e8da3$5496439d@news.astraweb.com>

On Tue, 22 Nov 2016 05:43 am, BartC wrote:

> The fastest I can get compiled, native code to do this is at 250 million
> cross-products per second.


Yes, yes, you're awfully clever, and your secret private language is so much
more efficient than even C that the entire IT industry ought to hang their
head in shame.

I'm only being *half* sarcastic here, for what its worth. I remember the
days when I could fit an entire operating system, plus applications, on a
400K floppy disk, and they would run at acceptable speed on something like
an 8 MHz CPU. Code used to be more efficient, with less overhead. But given
that your magic compiler runs only on one person's PC in the entire world,
it is completely irrelevant.



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Mon Nov 21 22:00:45 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Tue, 22 Nov 2016 14:00:45 +1100
Subject: Numpy slow at vector cross product?
References: <o0t22o$d9r$3@dont-email.me>
 <58326062$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <o0uo4r$gno$1@dont-email.me>
 <583309a9$0$1596$c3e8da3$5496439d@news.astraweb.com>
 <o107ve$3ad$1@dont-email.me>
Message-ID: <5833b4de$0$1616$c3e8da3$5496439d@news.astraweb.com>

On Tue, 22 Nov 2016 12:45 pm, BartC wrote:

> On 21/11/2016 14:50, Steve D'Aprano wrote:
>> On Mon, 21 Nov 2016 11:09 pm, BartC wrote:
> 
>> Modern machines run multi-tasking operating systems, where there can be
>> other processes running. Depending on what you use as your timer, you may
>> be measuring the time that those other processes run. The OS can cache
>> frequently used pieces of code, which allows it to run faster. The CPU
>> itself will cache some code.
> 
> You get to know after while what kinds of processes affect timings. For
> example, streaming a movie at the same time. 

Really, no.

You'll just have to take my word on this, but I'm not streaming any movies
at the moment. I don't even have a web browser running. And since I'm
running Linux, I don't have an anti-virus scanner that might have just
triggered a scan.

(But since I'm running Linux, I do have a web server, mail server, a DNS
server, cron, and about 300 other processes running, any of which might
start running for a microsecond or ten in the middle of a job.)

py> with Stopwatch():
...     x = math.sin(1.234)
...
elapsed time is very small; consider using the timeit module for
micro-timings of small code snippets
time taken: 0.007164 seconds


And again:

py> with Stopwatch():
...     x = math.sin(1.234)
...
elapsed time is very small; consider using the timeit module for
micro-timings of small code snippets
time taken: 0.000014 seconds


Look at the variation in the timing: 0.007164 versus 0.000014 second. That's
the influence of a cache, or more than one cache, somewhere. But if I run
it again:

py> with Stopwatch():
...     x = math.sin(1.234)
...
elapsed time is very small; consider using the timeit module for
micro-timings of small code snippets
time taken: 0.000013 seconds

there's a smaller variation, this time "only" 7%, for code which hasn't
changed. That's what your up against.

Running a whole lot of loops can, sometimes, mitigate some of that
variation, but not always. Even when running in a loop, you can easily get
variation of 10% or more just at random.


> So when you need to compare timings, you turn those off.
> 
>> The shorter the code snippet, the more these complications are relevant.
>> In this particular case, we can be reasonably sure that the time it takes
>> to create a list range(10000) and the overhead of the loop is *probably*
>> quite a small percentage of the time it takes to perform 100000 vector
>> multiplications. But that's not a safe assumption for all code snippets.
> 
> Yes, it was one of those crazy things that Python used to have to do,
> creating a list of N numbers just in order to be able to count to N.

Doesn't matter. Even with xrange, you're still counting the cost of looking
up xrange, passing one or more arguments to it, parsing those arguments,
creating an xrange object, and iterating over that xrange object
repeatedly. None of those things are free.

You might *hope* that the cost of those things are insignificant compared to
what you're actually interested in timing, but you don't know. And you're
resisting the idea of using a tool that is specifically designed to avoid
measuring all that overhead.

It's okay that your intuitions about the cost of executing Python code is
inaccurate. What's not okay is your refusal to listen to those who have a
better idea of what's involved.


[...]
>> The timeit module automates a bunch of tricky-to-right best practices for
>> timing code. Is that a problem?
> 
> The problem is it substitutes a bunch of tricky-to-get-right options and
> syntax which has to to typed /at the command line/. And you really don't
> want to have to write code at the command line (especially if sourced
> from elsewhere, which means you have to transcribe it).

You have to transcribe it no matter what you do. Unless you are given
correctly written timing code.

You don't have to use timeit from the command line. But you're mad if you
don't: the smaller the code snippet, the more convenient it is.

[steve at ando ~]$ python2.7 -m timeit -s "x = 257" "3*x"
10000000 loops, best of 3: 0.106 usec per loop
[steve at ando ~]$ python3.5 -m timeit -s "x = 257" "3*x"
10000000 loops, best of 3: 0.137 usec per loop


That's *brilliant* and much simpler than anything you are doing with loops
and clocks and whatnot. Its simple, straightforward, and tells me exactly
what I expected to see. (Python 3.6 will be even better.)

For the record, the reason Python 3.5 is so much slower here is because it
is a debugging build.


>> But if you prefer doing it "old school" from within Python, then:
>>
>> from timeit import Timer
>> t = Timer('np.cross(x, y)',  setup="""
>> import numpy as np
>> x = np.array([1, 2, 3])
>> y = np.array([4, 5, 6])
>> """)
>>
>> # take five measurements of 100000 calls each, and report the fastest
>> result = min(t.repeat(number=100000, repeat=5))/100000
>> print(result)  # time in seconds per call
> 
>> Better?
> 
> A bit, but the code is now inside a string!

As opposed to source code, which is... a string.


> Code will normally exist as a proper part of a module, not on the
> command line, in a command history, or in a string, so why not test it
> running inside a module?

Sure, you can do that, if you want potentially inaccurate results.


> But I've done a lot of benchmarking and actually measuring execution
> time is just part of it. This test I ran from inside a function for
> example, not at module-level, as that is more typical.
> 
> Are the variables inside a time-it string globals or locals? It's just a
> lot of extra factors to worry about, and extra things to get wrong.
> 
> The loop timings used by the OP showed one took considerably longer than
> the other. And that was confirmed by others. There's nothing wrong with
> that method.

In this specific example, the OP is comparing two radically different pieces
of code that clearly and obviously perform differently. He's doing the
equivalent of timing the code with his heartbeat, and getting 50 beats for
one and 150 beats for the other. That's good enough to show gross
differences in performance.

But often you're comparing two code snippets which are very nearly the same,
and trying to tease out a real difference of (say) 3% out of a noisy signal
where each run may differ by 10% just from randomness. Using your heartbeat
to time code is not going to do it.



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From rmjbros3 at gmail.com  Mon Nov 21 22:10:34 2016
From: rmjbros3 at gmail.com (rmjbros3 at gmail.com)
Date: Mon, 21 Nov 2016 19:10:34 -0800 (PST)
Subject: Can somebody tell me what's wrong wrong with my code? I don't
 understand
Message-ID: <e08ef692-5f20-494b-bb3e-676a6423ccf6@googlegroups.com>

Hi! This is my first post! I'm having trouble understanding my code. I get "SyntaxError:invalid syntax" on line 49. I'm trying to code a simple text-based rpg on repl.it. Thank you for reading.



print("Welcome to Gladiator Game! Choose your character race, class, and starting equipment!")

print('')

print("Race selection: ")

print('')

print("(1) Orcs. Known for their very wide, robust physiques. They are the strongest of all the races in Polaris.")

print('')

print("(2) Elves. Thin and wiry. They are known for their amazing agility and hand-eye coordiation. They originate from the desert island of Angolia.")

print('')

print("(3) Silverbacks. A hairy, ape-like race from Nothern Polaris. Their metal fur provides them with much needed protection.")

print('')

print("(4) Pomongos. An amphibian race believed to inhabit the wet jungles of Central Polaris. Legends say they have highly corrosive spit...")

print('')

raceNum=int(input("Select your character's race by entering the corresponding number. Then press enter: "))

print('')

while raceNum<1 or raceNum>4:
  raceNum=int(input('Invalid input. Try again: '))

print('')

if raceNum==1:
  print("You're an orc, eh? I won't be sayin' anything mean about you...")
  print('')
  classNum=int(input("What's your profession big fella?"))

elif raceNum==2:
  print("I never liked you elven folk...Let's get on with this.")
  print('')
  classNum=int(input("What's your profession ? Do ye even have one ?"))

elif raceNum==3:
  print("Nice fur. I don't see too many of your kind 'round here. Maybe that's a good thing...")
  print('')
  classNum=int(input("What's your profession mate?")
  
elif raceNum==4:             #this line has an error for some reason
  print("Your a 'Mongo eh? I thought you lads were extinct...Just keep your tongue in ya mouth and we'll get along fine mate.") 
  classNum=int(input("What's your profession?"))


From larry.martell at gmail.com  Mon Nov 21 22:18:14 2016
From: larry.martell at gmail.com (Larry Martell)
Date: Mon, 21 Nov 2016 22:18:14 -0500
Subject: Can somebody tell me what's wrong wrong with my code? I don't
 understand
In-Reply-To: <e08ef692-5f20-494b-bb3e-676a6423ccf6@googlegroups.com>
References: <e08ef692-5f20-494b-bb3e-676a6423ccf6@googlegroups.com>
Message-ID: <CACwCsY7GtO-xUtfXjBZ=kDNtGFicgPssN_Va1cR5k8xw3cbKAA@mail.gmail.com>

On Mon, Nov 21, 2016 at 10:10 PM,  <rmjbros3 at gmail.com> wrote:
> Hi! This is my first post! I'm having trouble understanding my code. I get "SyntaxError:invalid syntax" on line 49. I'm trying to code a simple text-based rpg on repl.it. Thank you for reading.
>
>
>
> print("Welcome to Gladiator Game! Choose your character race, class, and starting equipment!")
>
> print('')
>
> print("Race selection: ")
>
> print('')
>
> print("(1) Orcs. Known for their very wide, robust physiques. They are the strongest of all the races in Polaris.")
>
> print('')
>
> print("(2) Elves. Thin and wiry. They are known for their amazing agility and hand-eye coordiation. They originate from the desert island of Angolia.")
>
> print('')
>
> print("(3) Silverbacks. A hairy, ape-like race from Nothern Polaris. Their metal fur provides them with much needed protection.")
>
> print('')
>
> print("(4) Pomongos. An amphibian race believed to inhabit the wet jungles of Central Polaris. Legends say they have highly corrosive spit...")
>
> print('')
>
> raceNum=int(input("Select your character's race by entering the corresponding number. Then press enter: "))
>
> print('')
>
> while raceNum<1 or raceNum>4:
>   raceNum=int(input('Invalid input. Try again: '))
>
> print('')
>
> if raceNum==1:
>   print("You're an orc, eh? I won't be sayin' anything mean about you...")
>   print('')
>   classNum=int(input("What's your profession big fella?"))
>
> elif raceNum==2:
>   print("I never liked you elven folk...Let's get on with this.")
>   print('')
>   classNum=int(input("What's your profession ? Do ye even have one ?"))
>
> elif raceNum==3:
>   print("Nice fur. I don't see too many of your kind 'round here. Maybe that's a good thing...")
>   print('')
>   classNum=int(input("What's your profession mate?")
>
> elif raceNum==4:             #this line has an error for some reason
>   print("Your a 'Mongo eh? I thought you lads were extinct...Just keep your tongue in ya mouth and we'll get along fine mate.")
>   classNum=int(input("What's your profession?"))

Hint: look on line 47


From rosuav at gmail.com  Mon Nov 21 22:20:40 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 22 Nov 2016 14:20:40 +1100
Subject: Can somebody tell me what's wrong wrong with my code? I don't
 understand
In-Reply-To: <e08ef692-5f20-494b-bb3e-676a6423ccf6@googlegroups.com>
References: <e08ef692-5f20-494b-bb3e-676a6423ccf6@googlegroups.com>
Message-ID: <CAPTjJmpizEEJim815S=vufAYGMLqcNJU_h754Fu4ZrvUHF0kbw@mail.gmail.com>

On Tue, Nov 22, 2016 at 2:10 PM,  <rmjbros3 at gmail.com> wrote:
> Hi! This is my first post! I'm having trouble understanding my code. I get "SyntaxError:invalid syntax" on line 49. I'm trying to code a simple text-based rpg on repl.it. Thank you for reading.
>
>
> elif raceNum==3:
>   print("Nice fur. I don't see too many of your kind 'round here. Maybe that's a good thing...")
>   print('')
>   classNum=int(input("What's your profession mate?")
>
> elif raceNum==4:             #this line has an error for some reason
>   print("Your a 'Mongo eh? I thought you lads were extinct...Just keep your tongue in ya mouth and we'll get along fine mate.")
>   classNum=int(input("What's your profession?"))

Welcome to the community! I've trimmed your code to highlight the part
I'm about to refer to.

One of the tricks to understanding these kinds of errors is knowing
how the code is read, which is: top to bottom, left to right, exactly
the same as in English. Sometimes, a problem with one line of code is
actually discovered on the next line of code. (Occasionally further
down.) When you get a syntax error at the beginning of a line, it's
worth checking the previous line to see if it's somehow unfinished.

Have a look at your two blocks of code here. See if you can spot a
difference. There is one, and it's causing your error.

I'm hinting rather than overtly pointing it out, so you get a chance
to try this for yourself. Have at it!

ChrisA


From steve+comp.lang.python at pearwood.info  Tue Nov 22 00:14:14 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Tue, 22 Nov 2016 16:14:14 +1100
Subject: Numpy slow at vector cross product?
References: <o0t22o$d9r$3@dont-email.me>
 <58326062$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <o0uo4r$gno$1@dont-email.me>
 <583309a9$0$1596$c3e8da3$5496439d@news.astraweb.com>
 <o107ve$3ad$1@dont-email.me>
 <5833b4de$0$1616$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <5833d428$0$11090$c3e8da3@news.astraweb.com>

On Tuesday 22 November 2016 14:00, Steve D'Aprano wrote:

> Running a whole lot of loops can, sometimes, mitigate some of that
> variation, but not always. Even when running in a loop, you can easily get
> variation of 10% or more just at random.

I think that needs to be emphasised: there's a lot of random noise in these 
measurements.

For big, heavyweight functions that do a lot of work, the noise is generally a 
tiny proportion, and you can safely ignore it. (At least for CPU bound tasks: 
I/O bound tasks, the noise in I/O is potentially very high.)

For really tiny operations, the noise *may* be small, depending on the 
operation.  But small is not insignificant. Consider a simple operation like 
addition:

# Python 3.5
import statistics
from timeit import Timer
t = Timer("x + 1", setup="x = 0")
# ten trials, of one million loops each
results = t.repeat(repeat=10)
best = min(results)
average = statistics.mean(results)
std_error = statistics.stdev(results)/statistics.mean(results)


Best: 0.09761243686079979
Average: 0.0988507878035307
Std error: 0.02260956789268462

So this suggests that on my machine, doing no expensive virus scans or 
streaming video, the random noise in something as simple as integer addition is 
around two percent.

So that's your baseline: even simple operations repeated thousands of times 
will show random noise of a few percent.

Consequently, if you're doing one trial (one loop of, say, a million 
operations):

start = time.time()
for i in range(1000000):
    x + 1
elapsed = time.time() - start


and compare the time taken with another trial, and the difference is of the 
order of a few percentage points, then you have *no* reason to believe the 
result is real. You ought to repeat your test multiple times -- the more the 
better.

timeit makes it easy to repeat your tests. It automatically picks the best 
timer for your platform and avoid serious gotchas from using the wrong timer. 
When called from the command line, it will automatically select the best number 
of loops to ensure reliable timing, without wasting time doing more loops than 
needed.

timeit isn't magic. It's not doing anything that you or I couldn't do by hand, 
if we knew we should be doing it, and if we could be bothered to run multiple 
trials and gather statistics and keep a close eye on the deviation between 
measurements. But who wants to do that by hand?



-- 
Steven
299792.458 km/s ? not just a good idea, it?s the law!



From steve+comp.lang.python at pearwood.info  Tue Nov 22 00:19:54 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Tue, 22 Nov 2016 16:19:54 +1100
Subject: Can somebody tell me what's wrong wrong with my code? I don't
 understand
References: <e08ef692-5f20-494b-bb3e-676a6423ccf6@googlegroups.com>
Message-ID: <5833d57a$0$11090$c3e8da3@news.astraweb.com>

On Tuesday 22 November 2016 14:10, rmjbros3 at gmail.com wrote:

> Hi! This is my first post! I'm having trouble understanding my code. I get
> "SyntaxError:invalid syntax" on line 49.

Sometimes a syntax error can only be reported on the line *following* the line 
with the actual error. So you may have something like this:


x = func(y  # oops forgot to close the brackets
for i in range(x):  # error is reported here
    ...


So when you get a syntax error on a line, and cannot see anything wrong with 
that line, work your way backwards until you find the actually location of the 
error.


[...]
> elif raceNum==3:
>   print("Nice fur. I don't see too many of your kind 'round here. Maybe
>   that's a good thing...") 
>   print('')
>   classNum=int(input("What's your profession mate?")
>   
> elif raceNum==4:             #this line has an error for some reason

This is the line where the error is reported. The error is actually on the 
previous line. Do you see it now?





-- 
Steven
299792.458 km/s ? not just a good idea, it?s the law!



From no.email at nospam.invalid  Tue Nov 22 00:30:20 2016
From: no.email at nospam.invalid (Paul Rubin)
Date: Mon, 21 Nov 2016 21:30:20 -0800
Subject: Numpy slow at vector cross product?
References: <o0t22o$d9r$3@dont-email.me>
 <58326062$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <o0uo4r$gno$1@dont-email.me>
 <583309a9$0$1596$c3e8da3$5496439d@news.astraweb.com>
 <o107ve$3ad$1@dont-email.me>
 <5833b4de$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <5833d428$0$11090$c3e8da3@news.astraweb.com>
Message-ID: <87vavg9j4j.fsf@nightsong.com>

Steven D'Aprano <steve+comp.lang.python at pearwood.info> writes:
> if we knew we should be doing it, and if we could be bothered to run
> multiple trials and gather statistics and keep a close eye on the
> deviation between measurements. But who wants to do that by hand?

You might like this, for Haskell:

   http://www.serpentine.com/criterion/tutorial.html

I've sometimes thought of wrapping it around other languages.

Make sure to click on the graphs: it's impressive work.


From nad at python.org  Tue Nov 22 02:02:51 2016
From: nad at python.org (Ned Deily)
Date: Tue, 22 Nov 2016 02:02:51 -0500
Subject: [RELEASE] Python 3.6.0b4 is now available
Message-ID: <FA2D757B-85AE-4208-ADD2-41AF790C6032@python.org>

On behalf of the Python development community and the Python 3.6 release
team, I'm pleased to announce the availability of Python 3.6.0b4. 3.6.0b4
is the last planned beta release of Python 3.6, the next major release of
Python.

Among the new major new features in Python 3.6 are:

* PEP 468 - Preserving the order of **kwargs in a function
* PEP 487 - Simpler customization of class creation
* PEP 495 - Local Time Disambiguation
* PEP 498 - Literal String Formatting
* PEP 506 - Adding A Secrets Module To The Standard Library
* PEP 509 - Add a private version to dict
* PEP 515 - Underscores in Numeric Literals
* PEP 519 - Adding a file system path protocol
* PEP 520 - Preserving Class Attribute Definition Order
* PEP 523 - Adding a frame evaluation API to CPython
* PEP 524 - Make os.urandom() blocking on Linux (during system startup)
* PEP 525 - Asynchronous Generators (provisional)
* PEP 526 - Syntax for Variable Annotations (provisional)
* PEP 528 - Change Windows console encoding to UTF-8
* PEP 529 - Change Windows filesystem encoding to UTF-8
* PEP 530 - Asynchronous Comprehensions

Please see "What?s New In Python 3.6" for more information:

https://docs.python.org/3.6/whatsnew/3.6.html

You can find Python 3.6.0b4 here:

https://www.python.org/downloads/release/python-360b4/

Beta releases are intended to give the wider community the opportunity
to test new features and bug fixes and to prepare their projects to
support the new feature release. We strongly encourage maintainers of
third-party Python projects to test with 3.6 during the beta phase and
report issues found to bugs.python.org as soon as possible. While the
release is feature complete entering the beta phase, it is possible that
features may be modified or, in rare cases, deleted up until the start
of the release candidate phase (2016-12-05). Our goal is have no changes
after rc1. To achieve that, it will be extremely important to get as
much exposure for 3.6 as possible during the beta phase. Please keep in
mind that this is a preview release and its use is not recommended for
production environments

The next pre-release of Python 3.6 will be 3.6.0rc1, the release candidate,
currently scheduled for 2016-12-05. The official release of Python 3.6.0
is currently scheduled for 2016-12-16.  More information about the release
schedule can be found here:

https://www.python.org/dev/peps/pep-0494/

--
  Ned Deily
  nad at python.org -- []



From gheskett at shentel.net  Tue Nov 22 02:43:34 2016
From: gheskett at shentel.net (Gene Heskett)
Date: Tue, 22 Nov 2016 02:43:34 -0500
Subject: Can somebody tell me what's wrong wrong with my code? I don't
 understand
In-Reply-To: <CAPTjJmpizEEJim815S=vufAYGMLqcNJU_h754Fu4ZrvUHF0kbw@mail.gmail.com>
References: <e08ef692-5f20-494b-bb3e-676a6423ccf6@googlegroups.com>
 <CAPTjJmpizEEJim815S=vufAYGMLqcNJU_h754Fu4ZrvUHF0kbw@mail.gmail.com>
Message-ID: <201611220243.34250.gheskett@shentel.net>

On Monday 21 November 2016 22:20:40 Chris Angelico wrote:

> On Tue, Nov 22, 2016 at 2:10 PM,  <rmjbros3 at gmail.com> wrote:
> > Hi! This is my first post! I'm having trouble understanding my code.
> > I get "SyntaxError:invalid syntax" on line 49. I'm trying to code a
> > simple text-based rpg on repl.it. Thank you for reading.
> >
> >
> > elif raceNum==3:
> >   print("Nice fur. I don't see too many of your kind 'round here.
> > Maybe that's a good thing...") print('')
> >   classNum=int(input("What's your profession mate?")
> >
> > elif raceNum==4:             #this line has an error for some reason
> >   print("Your a 'Mongo eh? I thought you lads were extinct...Just
> > keep your tongue in ya mouth and we'll get along fine mate.")
> > classNum=int(input("What's your profession?"))
>
> Welcome to the community! I've trimmed your code to highlight the part
> I'm about to refer to.
>
> One of the tricks to understanding these kinds of errors is knowing
> how the code is read, which is: top to bottom, left to right, exactly
> the same as in English. Sometimes, a problem with one line of code is
> actually discovered on the next line of code. (Occasionally further
> down.) When you get a syntax error at the beginning of a line, it's
> worth checking the previous line to see if it's somehow unfinished.
>
> Have a look at your two blocks of code here. See if you can spot a
> difference. There is one, and it's causing your error.
>
> I'm hinting rather than overtly pointing it out, so you get a chance
> to try this for yourself. Have at it!
>
> ChrisA

I'm a fading 82 yo, and python dummy, but I think I see it.  In fact, I 
wrote a program to check for that and similar errors in my C code back 
in the late '80's. I called it cntx at the time.

Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>


From prihantoro2001 at gmail.com  Tue Nov 22 03:18:54 2016
From: prihantoro2001 at gmail.com (prihantoro2001 at gmail.com)
Date: Tue, 22 Nov 2016 00:18:54 -0800 (PST)
Subject: Result is not Displayed
Message-ID: <729cb952-0f05-462a-b07d-e0e60cfbb1aa@googlegroups.com>

Dear all, 

i am new to Python and have this problem

=====
import nltk
puzzle_letters = nltk.FreqDist('egivrvonl')
obligatory = 'r'
wordlist = nltk.corpus.words.words()
[w for w in wordlist if len(w) >= 6
and obligatory in w
and nltk.FreqDist(w) <= puzzle_letters]
print puzzle_letters
======

this gives me <FreqDist with 8 samples and 9 outcomes>
while the expected outcome is ['glover', 'govern', ...]
did i miss something?

Thank you

Pri


From irmen.NOSPAM at xs4all.nl  Tue Nov 22 03:52:51 2016
From: irmen.NOSPAM at xs4all.nl (Irmen de Jong)
Date: Tue, 22 Nov 2016 09:52:51 +0100
Subject: Result is not Displayed
In-Reply-To: <729cb952-0f05-462a-b07d-e0e60cfbb1aa@googlegroups.com>
References: <729cb952-0f05-462a-b07d-e0e60cfbb1aa@googlegroups.com>
Message-ID: <58340763$0$21406$e4fe514c@news.xs4all.nl>

On 22-11-2016 9:18, prihantoro2001 at gmail.com wrote:
> Dear all, 
> 
> i am new to Python and have this problem
> 
> =====
> import nltk
> puzzle_letters = nltk.FreqDist('egivrvonl')
> obligatory = 'r'
> wordlist = nltk.corpus.words.words()
> [w for w in wordlist if len(w) >= 6
> and obligatory in w
> and nltk.FreqDist(w) <= puzzle_letters]
> print puzzle_letters
> ======
> 
> this gives me <FreqDist with 8 samples and 9 outcomes>
> while the expected outcome is ['glover', 'govern', ...]
> did i miss something?


Review your code carefully. You're printing the wrong thing at the end, and you're doing
nothing with the list comprehension that comes before it.

-irmen



From __peter__ at web.de  Tue Nov 22 03:55:17 2016
From: __peter__ at web.de (Peter Otten)
Date: Tue, 22 Nov 2016 09:55:17 +0100
Subject: Result is not Displayed
References: <729cb952-0f05-462a-b07d-e0e60cfbb1aa@googlegroups.com>
Message-ID: <o1115i$ikl$1@blaine.gmane.org>

prihantoro2001 at gmail.com wrote:

> Dear all,
> 
> i am new to Python and have this problem
> 
> =====
> import nltk
> puzzle_letters = nltk.FreqDist('egivrvonl')
> obligatory = 'r'
> wordlist = nltk.corpus.words.words()
> [w for w in wordlist if len(w) >= 6
> and obligatory in w
> and nltk.FreqDist(w) <= puzzle_letters]
> print puzzle_letters
> ======
> 
> this gives me <FreqDist with 8 samples and 9 outcomes>
> while the expected outcome is ['glover', 'govern', ...]
> did i miss something?

You asked for letters when you wanted to see the words. You actually build 
the list of words with

> [w for w in wordlist if len(w) >= 6
> and obligatory in w
> and nltk.FreqDist(w) <= puzzle_letters]

but don't even bind it to a name. Try

words = [
    w for w in wordlist if len(w) >= 6
    and obligatory in w
    and nltk.FreqDist(w) <= puzzle_letters
]
print words

to get what you want.



From bc at freeuk.com  Tue Nov 22 06:47:05 2016
From: bc at freeuk.com (BartC)
Date: Tue, 22 Nov 2016 11:47:05 +0000
Subject: Numpy slow at vector cross product?
In-Reply-To: <5833b121$0$1603$c3e8da3$5496439d@news.astraweb.com>
References: <o0t22o$d9r$3@dont-email.me>
 <58326062$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <o0uq77$cjf$1@blaine.gmane.org>
 <mailman.18.1479732277.3146.python-list@python.org>
 <o0v1oe$jju$1@dont-email.me> <pan.2016.11.21.17.04.32.390000@nowhere.invalid>
 <o0vf84$ble$1@dont-email.me>
 <5833b121$0$1603$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o11b6o$rfr$1@dont-email.me>

On 22/11/2016 02:44, Steve D'Aprano wrote:
> On Tue, 22 Nov 2016 05:43 am, BartC wrote:
>
>> The fastest I can get compiled, native code to do this is at 250 million
>> cross-products per second.

(Actually 300 million using 64-bit code.)

> Yes, yes, you're awfully clever, and your secret private language is so much
> more efficient than even C that the entire IT industry ought to hang their
> head in shame.

The 250M/300M timing was using C and gcc-O3.

This gives an indication of the upper limit of what is possible.

The suggestion was that the numpy solution, which was *one thousand time 
slower* than these figures, wouldn't be any faster if written in C.

I'm simply suggesting there is plenty of room for improvement. I even 
showed a version that did *exactly* what numpy does (AFAIK) that was 
three times the speed of numpy even executed by CPython. So there is 
some mystery there.

(FWIW my own compiled language manages 70M, and my interpreted language 
up to 3M. Poor, so I might look at them again next time I have to do 
loads of vector-products.)

> I'm only being *half* sarcastic here, for what its worth. I remember the
> days when I could fit an entire operating system, plus applications, on a
> 400K floppy disk, and they would run at acceptable speed on something like
> an 8 MHz CPU. Code used to be more efficient, with less overhead. But given
> that your magic compiler runs only on one person's PC in the entire world,
> it is completely irrelevant.

So you're saying that numpy's speed is perfectly reasonable and we 
should do nothing about it? Because, after all, it does a few extra 
clever things (whether the programmer wants them or not!).

And the way it is written is completely beyond reproach:

     msg = "incompatible dimensions for cross product\n"\
           "(dimension must be 2 or 3)"
     if (a.shape[0] not in [2, 3]) or (b.shape[0] not in [2, 3]):
         raise ValueError(msg)

So it is fine for msg to be bound to a string value here (and having to 
unbind it again when it returns) even though it will never be used again 
in a normal call. (Or having to test both a.shape[0] and b.shape[0] for 
inclusion in a list, expensive looking operations, when they are tested 
again anyway in the next few lines.)

-- 
Bartc


From skip.montanaro at gmail.com  Tue Nov 22 07:34:49 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Tue, 22 Nov 2016 06:34:49 -0600
Subject: Numpy slow at vector cross product?
In-Reply-To: <o11b6o$rfr$1@dont-email.me>
References: <o0t22o$d9r$3@dont-email.me>
 <58326062$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <o0uq77$cjf$1@blaine.gmane.org>
 <mailman.18.1479732277.3146.python-list@python.org>
 <o0v1oe$jju$1@dont-email.me> <pan.2016.11.21.17.04.32.390000@nowhere.invalid>
 <o0vf84$ble$1@dont-email.me>
 <5833b121$0$1603$c3e8da3$5496439d@news.astraweb.com>
 <o11b6o$rfr$1@dont-email.me>
Message-ID: <CANc-5UzR27T4fkP_JodSNkp32wz1dwnePMHnjGdtwvffdHhd8w@mail.gmail.com>

> I'm simply suggesting there is plenty of room for improvement. I even
showed a version that did *exactly* what numpy does (AFAIK) that was three
times the speed of numpy even executed by CPython. So there is some mystery
there.

As I indicated in my earlier response, your version doesn't pass all of
numpy's cross product unit tests. Fix that and submit a patch to the numpy
maintainers. I suspect it would be accepted.

Skip


From bc at freeuk.com  Tue Nov 22 07:45:12 2016
From: bc at freeuk.com (BartC)
Date: Tue, 22 Nov 2016 12:45:12 +0000
Subject: Numpy slow at vector cross product?
In-Reply-To: <mailman.34.1479818099.3146.python-list@python.org>
References: <o0t22o$d9r$3@dont-email.me>
 <58326062$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <o0uq77$cjf$1@blaine.gmane.org>
 <mailman.18.1479732277.3146.python-list@python.org>
 <o0v1oe$jju$1@dont-email.me> <pan.2016.11.21.17.04.32.390000@nowhere.invalid>
 <o0vf84$ble$1@dont-email.me>
 <5833b121$0$1603$c3e8da3$5496439d@news.astraweb.com>
 <o11b6o$rfr$1@dont-email.me>
 <CANc-5UzR27T4fkP_JodSNkp32wz1dwnePMHnjGdtwvffdHhd8w@mail.gmail.com>
 <mailman.34.1479818099.3146.python-list@python.org>
Message-ID: <o11ejn$7i2$1@dont-email.me>

On 22/11/2016 12:34, Skip Montanaro wrote:
>> I'm simply suggesting there is plenty of room for improvement. I even
> showed a version that did *exactly* what numpy does (AFAIK) that was three
> times the speed of numpy even executed by CPython. So there is some mystery
> there.
>
> As I indicated in my earlier response, your version doesn't pass all of
> numpy's cross product unit tests. Fix that and submit a patch to the numpy
> maintainers. I suspect it would be accepted.

I saw your response but didn't understand it. My code was based around 
what Peter Otten posted from numpy sources.

I will have a look. Don't forget however that all someone is trying to 
do is to multiply two vectors. They're not interested in axes 
transformation or making them broadcastable, whatever that means.

So making numpy.cross do all that may simply be too big a cost.

-- 
bartc



From bc at freeuk.com  Tue Nov 22 08:06:05 2016
From: bc at freeuk.com (BartC)
Date: Tue, 22 Nov 2016 13:06:05 +0000
Subject: Numpy slow at vector cross product?
In-Reply-To: <5833b4de$0$1616$c3e8da3$5496439d@news.astraweb.com>
References: <o0t22o$d9r$3@dont-email.me>
 <58326062$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <o0uo4r$gno$1@dont-email.me>
 <583309a9$0$1596$c3e8da3$5496439d@news.astraweb.com>
 <o107ve$3ad$1@dont-email.me>
 <5833b4de$0$1616$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o11fqs$bfq$1@dont-email.me>

On 22/11/2016 03:00, Steve D'Aprano wrote:
> On Tue, 22 Nov 2016 12:45 pm, BartC wrote:

>> You get to know after while what kinds of processes affect timings. For
>> example, streaming a movie at the same time.
>
> Really, no.

> py> with Stopwatch():
> ...     x = math.sin(1.234)
> ...
> elapsed time is very small; consider using the timeit module for
> micro-timings of small code snippets
> time taken: 0.007164 seconds

I tried 'Stopwatch()' and it didn't understand what it was.

> And again:
>
> py> with Stopwatch():
> ...     x = math.sin(1.234)
> ...
> elapsed time is very small; consider using the timeit module for
> micro-timings of small code snippets
> time taken: 0.000014 seconds
>
>
> Look at the variation in the timing: 0.007164 versus 0.000014 second. That's
> the influence of a cache, or more than one cache, somewhere. But if I run
> it again:
>
> py> with Stopwatch():
> ...     x = math.sin(1.234)
> ...
> elapsed time is very small; consider using the timeit module for
> micro-timings of small code snippets
> time taken: 0.000013 seconds
>
> there's a smaller variation, this time "only" 7%, for code which hasn't
> changed. That's what your up against.

I tried this:

import math

def fn():
	for i in xrange(3000000):
		x=math.sin(1.234)
	print x

fn()

If I run this, my IDE tells me the whole thing took 0.93 seconds. That's 
including the overheads of the IDE, invoking python.exe, Python's 
start-up time, the function call, the loop overheads, and whatever is 
involved in shutting it all down again.

Now I replace the x=math.sin() line with pass. The IDE now says 0.21 
seconds. The only thing that's changed is the sin() call. I can probably 
deduce that executing x=math.sin(1.2340) three million times took 0.72 
seconds (with some expected variation).

So over 4 million times per second. Which is interesting because your 
timings with Stopwatch varied from 140 to 71000 per second (and I doubt 
your machine is that much slower than mine).


> [steve at ando ~]$ python2.7 -m timeit -s "x = 257" "3*x"
> 10000000 loops, best of 3: 0.106 usec per loop
> [steve at ando ~]$ python3.5 -m timeit -s "x = 257" "3*x"
> 10000000 loops, best of 3: 0.137 usec per loop

> That's *brilliant* and much simpler than anything you are doing with loops
> and clocks and whatnot.

But you lose all context. And there is a limit to what you can do with 
such micro-benchmarks.

Sometimes you want to time a real task, but with one line substituted 
for another, or some other minor rearrangement.

Or perhaps the overall time of a code fragment depends on the mix of 
data it has to deal with.

>> Code will normally exist as a proper part of a module, not on the
>> command line, in a command history, or in a string, so why not test it
>> running inside a module?
>
> Sure, you can do that, if you want potentially inaccurate results.

When your customer runs your application (and complains about how long 
it takes), they will be running the code as normal not inside a string 
or on the command line!

> In this specific example, the OP is comparing two radically different pieces
> of code that clearly and obviously perform differently. He's doing the
> equivalent of timing the code with his heartbeat, and getting 50 beats for
> one and 150 beats for the other. That's good enough to show gross
> differences in performance.

No, he's using time.clock() with, presumably, a consistent number of 
ticks per second.

> But often you're comparing two code snippets which are very nearly the same,
> and trying to tease out a real difference of (say) 3% out of a noisy signal
> where each run may differ by 10% just from randomness. Using your heartbeat
> to time code is not going to do it.

Yes, exactly, that's why typing a code fragment on the command line is a 
waste of time. You need to design a test where the differences are going 
to be more obvious.

-- 
Bartc


From eryksun at gmail.com  Tue Nov 22 08:42:55 2016
From: eryksun at gmail.com (eryk sun)
Date: Tue, 22 Nov 2016 13:42:55 +0000
Subject: Numpy slow at vector cross product?
In-Reply-To: <o11fqs$bfq$1@dont-email.me>
References: <o0t22o$d9r$3@dont-email.me>
 <58326062$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <o0uo4r$gno$1@dont-email.me>
 <583309a9$0$1596$c3e8da3$5496439d@news.astraweb.com>
 <o107ve$3ad$1@dont-email.me>
 <5833b4de$0$1616$c3e8da3$5496439d@news.astraweb.com>
 <o11fqs$bfq$1@dont-email.me>
Message-ID: <CACL+1ataCTQd22rP1BwuDHs37w=-XNwUvptuH5OK3o4eR1L_yQ@mail.gmail.com>

On Tue, Nov 22, 2016 at 1:06 PM, BartC <bc at freeuk.com> wrote:
>> In this specific example, the OP is comparing two radically different
>> pieces of code that clearly and obviously perform differently. He's doing
>> the equivalent of timing the code with his heartbeat, and getting 50 beats
>> for one and 150 beats for the other. That's good enough to show gross
>> differences in performance.
>
> No, he's using time.clock() with, presumably, a consistent number of ticks
> per second.

Note that some people in this discussion use Unix systems, on which
using time.clock for wall-clock timing is completely wrong. Please use
timeit.default_timer to ensure that examples are portable.


From bc at freeuk.com  Tue Nov 22 09:45:06 2016
From: bc at freeuk.com (BartC)
Date: Tue, 22 Nov 2016 14:45:06 +0000
Subject: Numpy slow at vector cross product?
In-Reply-To: <o11ejn$7i2$1@dont-email.me>
References: <o0t22o$d9r$3@dont-email.me>
 <58326062$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <o0uq77$cjf$1@blaine.gmane.org>
 <mailman.18.1479732277.3146.python-list@python.org>
 <o0v1oe$jju$1@dont-email.me> <pan.2016.11.21.17.04.32.390000@nowhere.invalid>
 <o0vf84$ble$1@dont-email.me>
 <5833b121$0$1603$c3e8da3$5496439d@news.astraweb.com>
 <o11b6o$rfr$1@dont-email.me>
 <CANc-5UzR27T4fkP_JodSNkp32wz1dwnePMHnjGdtwvffdHhd8w@mail.gmail.com>
 <mailman.34.1479818099.3146.python-list@python.org>
 <o11ejn$7i2$1@dont-email.me>
Message-ID: <o11lkh$1nk$1@dont-email.me>

On 22/11/2016 12:45, BartC wrote:
> On 22/11/2016 12:34, Skip Montanaro wrote:
>>> I'm simply suggesting there is plenty of room for improvement. I even
>> showed a version that did *exactly* what numpy does (AFAIK) that was
>> three
>> times the speed of numpy even executed by CPython. So there is some
>> mystery
>> there.
>>
>> As I indicated in my earlier response, your version doesn't pass all of
>> numpy's cross product unit tests. Fix that and submit a patch to the
>> numpy
>> maintainers. I suspect it would be accepted.
>
> I saw your response but didn't understand it. My code was based around
> what Peter Otten posted from numpy sources.
>
> I will have a look. Don't forget however that all someone is trying to
> do is to multiply two vectors. They're not interested in axes
> transformation or making them broadcastable, whatever that means.

It seems the posted code of numpy.cross wasn't complete. The full code 
is below (I've removed the doc string, but have had to add some 'np.' 
prefixes as it is now out of context).

If anyone is still wondering why numpy.cross is slow, then no further 
comments are necessary!

----------------------------------------------
def cross(a, b, axisa=-1, axisb=-1, axisc=-1, axis=None):

     if axis is not None:
         axisa, axisb, axisc = (axis,) * 3
     a = np.asarray(a)
     b = np.asarray(b)
     # Check axisa and axisb are within bounds
     axis_msg = "'axis{0}' out of bounds"
     if axisa < -a.ndim or axisa >= a.ndim:
         raise ValueError(axis_msg.format('a'))
     if axisb < -b.ndim or axisb >= b.ndim:
         raise ValueError(axis_msg.format('b'))
     # Move working axis to the end of the shape
     a = np.rollaxis(a, axisa, a.ndim)
     b = np.rollaxis(b, axisb, b.ndim)
     msg = ("incompatible dimensions for cross product\n"
            "(dimension must be 2 or 3)")
     if a.shape[-1] not in (2, 3) or b.shape[-1] not in (2, 3):
         raise ValueError(msg)

     # Create the output array
     shape = np.broadcast(a[..., 0], b[..., 0]).shape
     if a.shape[-1] == 3 or b.shape[-1] == 3:
         shape += (3,)
         # Check axisc is within bounds
         if axisc < -len(shape) or axisc >= len(shape):
             raise ValueError(axis_msg.format('c'))
     dtype = np.promote_types(a.dtype, b.dtype)
     cp = np.empty(shape, dtype)

     # create local aliases for readability
     a0 = a[..., 0]
     a1 = a[..., 1]
     if a.shape[-1] == 3:
         a2 = a[..., 2]
     b0 = b[..., 0]
     b1 = b[..., 1]
     if b.shape[-1] == 3:
         b2 = b[..., 2]
     if cp.ndim != 0 and cp.shape[-1] == 3:
         cp0 = cp[..., 0]
         cp1 = cp[..., 1]
         cp2 = cp[..., 2]

     if a.shape[-1] == 2:
         if b.shape[-1] == 2:
             # a0 * b1 - a1 * b0
             multiply(a0, b1, out=cp)
             cp -= a1 * b0
             return cp
         else:
             assert b.shape[-1] == 3
             # cp0 = a1 * b2 - 0  (a2 = 0)
             # cp1 = 0 - a0 * b2  (a2 = 0)
             # cp2 = a0 * b1 - a1 * b0
             np.multiply(a1, b2, out=cp0)
             np.multiply(a0, b2, out=cp1)
             np.negative(cp1, out=cp1)
             np.multiply(a0, b1, out=cp2)
             cp2 -= a1 * b0
     else:
         assert a.shape[-1] == 3
         if b.shape[-1] == 3:
             # cp0 = a1 * b2 - a2 * b1
             # cp1 = a2 * b0 - a0 * b2
             # cp2 = a0 * b1 - a1 * b0
             np.multiply(a1, b2, out=cp0)
             tmp = np.array(a2 * b1)
             cp0 -= tmp
             np.multiply(a2, b0, out=cp1)
             np.multiply(a0, b2, out=tmp)
             cp1 -= tmp
             np.multiply(a0, b1, out=cp2)
             np.multiply(a1, b0, out=tmp)
             cp2 -= tmp
         else:
             assert b.shape[-1] == 2
             # cp0 = 0 - a2 * b1  (b2 = 0)
             # cp1 = a2 * b0 - 0  (b2 = 0)
             # cp2 = a0 * b1 - a1 * b0
             np.multiply(a2, b1, out=cp0)
             np.negative(cp0, out=cp0)
             np.multiply(a2, b0, out=cp1)
             np.multiply(a0, b1, out=cp2)
             cp2 -= a1 * b0

     # This works because we are moving the last axis
     return np.rollaxis(cp, -1, axisc)


-- 
Bartc


From ganesh1pal at gmail.com  Tue Nov 22 10:27:04 2016
From: ganesh1pal at gmail.com (Ganesh Pal)
Date: Tue, 22 Nov 2016 20:57:04 +0530
Subject: How to you convert list of tuples to string
Message-ID: <CACT3xuX0BM3txkDttjp8Nebn6DaDLUMyVQxuC0t3Mj27VHjc6w@mail.gmail.com>

Dear friends ,


I am using fedora 18 and on  Python 2.7 version


I have a list of tuples as shown below

>> list

[(1, 1, 373891072L, 8192), (1, 3, 390348800L, 8192), (1, 4, 372719616L,
8192), (2, 3, 382140416L, 8192), (2, 5, 398721024L, 8192), (3, 1,
374030336L, 8192), (3, 3, 374079488L, 8192), (3, 5, 340058112L, 8192)]

(a) I need to select any element randomly the list say (x, y, xxxxxxxxxL,
8192)

  >>> list
     [(1, 1, 373891072L, 8192), (1, 3, 390348800L, 8192), (1, 4,
372719616L, 8192), (2, 3, 382140416L, 8192), (2, 5, 398721024L, 8192), (3,
1, 374030336L, 8192), (3, 3, 374079488L, 8192), (3, 5, 340058112L, 8192)]


  >>> import random
  >>> i = random.randrange(len(list))
  >>> sel_item = list[i]
  >>> sel_item
  (3, 5, 340058112L, 8192)



(b) Then convert the selected item in the below format i.e
 1,1,373891072:8192 ( strip L and add :)

     >>> sel_item
  (3, 5, 340058112L, 8192)
   >> c1 = ','.join(map(str,sel_item))

# what happened to 'L' it got stripped automatically ? will these be a
problem
   >>> c1
       '3,5,340058112,8192'
#last four are always 8912 and
   >>> c1 = c1[0:-5] + ':8912'
    >>> c1
 '3,5,340058112:8912'
   >>>


Any better suggestion to improve this piece of code and make it look more /
pythonic


Regards,
Ganesh Pal


From steve+python at pearwood.info  Tue Nov 22 11:48:56 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Wed, 23 Nov 2016 03:48:56 +1100
Subject: Numpy slow at vector cross product?
References: <o0t22o$d9r$3@dont-email.me>
 <58326062$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <o0uq77$cjf$1@blaine.gmane.org>
 <mailman.18.1479732277.3146.python-list@python.org>
 <o0v1oe$jju$1@dont-email.me> <pan.2016.11.21.17.04.32.390000@nowhere.invalid>
 <o0vf84$ble$1@dont-email.me>
 <5833b121$0$1603$c3e8da3$5496439d@news.astraweb.com>
 <o11b6o$rfr$1@dont-email.me>
 <CANc-5UzR27T4fkP_JodSNkp32wz1dwnePMHnjGdtwvffdHhd8w@mail.gmail.com>
 <mailman.34.1479818099.3146.python-list@python.org>
 <o11ejn$7i2$1@dont-email.me>
Message-ID: <583476fa$0$1583$c3e8da3$5496439d@news.astraweb.com>

On Tue, 22 Nov 2016 11:45 pm, BartC wrote:


> I will have a look. Don't forget however that all someone is trying to
> do is to multiply two vectors. They're not interested in axes
> transformation or making them broadcastable, whatever that means.

You don't know that.

Bart, you have a rather disagreeable tendency towards assuming that if you
don't want to do something, then nobody else could possibly want to do it.
You should try to keep an open mind to the possibility that perhaps there
are use-cases that you didn't think of. numpy is not *the* most heavily
used third-party library in the Python ecosystem because its slow.

numpy is a library for doing vectorised operations over massive arrays.
You're thinking of numpy users doing the cross product of two vectors, but
you should be thinking of numpy users doing the cross product of a million
pairs of vectors.

Could numpy optimize the single pair of vectors case a bit better? Perhaps
they could. It looks like there's a bunch of minor improvements which could
be made to the code, a few micro-optimizations that shave a microsecond or
two off the execution time. And maybe they could even detect the case where
the arguments are a single pair of vectors, and optimize that. Even
replacing it with a naive pure-Python cross product would be a big win.

But for the big array of vectors case, you absolutely have to support doing
fast vectorized cross-products over a huge number of vectors.

py> a = np.array([
...     [1, 2, 3],
...     [4, 5, 6],
...     [7, 8, 9],
...     ]*1000
...     )
py> b = np.array([
...     [9, 8, 7],
...     [6, 5, 4],
...     [3, 2, 1],
...     ]*1000
...     )
py> a.shape
(3000, 3)
py> result = np.cross(a, b)
py> result.shape
(3000, 3)


On my computer, numpy took only 10 times longer to cross-multiply 3000 pairs
of vectors than it took to cross-multiply a single pair of vectors. If I
did that in pure Python, it would take 3000 times longer, or more, so numpy
wins here by a factor of 300.




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From motoom at xs4all.nl  Tue Nov 22 11:51:58 2016
From: motoom at xs4all.nl (Michiel Overtoom)
Date: Tue, 22 Nov 2016 17:51:58 +0100
Subject: How to you convert list of tuples to string
In-Reply-To: <CACT3xuX0BM3txkDttjp8Nebn6DaDLUMyVQxuC0t3Mj27VHjc6w@mail.gmail.com>
References: <CACT3xuX0BM3txkDttjp8Nebn6DaDLUMyVQxuC0t3Mj27VHjc6w@mail.gmail.com>
Message-ID: <CDC17C8A-463D-4247-B1BB-D1D46A6243E3@xs4all.nl>

Hi Ganesh,

> Any better suggestion to improve this piece of code and make it look more pythonic?


import random

# A list of tuples. Note that the L behind a number means that the number is a 'long'.

data = [(1, 1, 373891072L, 8192), (1, 3, 390348800L, 8192), (1, 4, 372719616L,
    8192), (2, 3, 382140416L, 8192), (2, 5, 398721024L, 8192), (3, 1,
    374030336L, 8192), (3, 3, 374079488L, 8192), (3, 5, 340058112L, 8192)]

item = random.choice(data)  # Select a random item from the 'data' list.

msg = "%d,%d,%d:%d" % item  # Format it in the way you like.

print msg


Greetings,



From jon+usenet at unequivocal.eu  Tue Nov 22 14:05:40 2016
From: jon+usenet at unequivocal.eu (Jon Ribbens)
Date: Tue, 22 Nov 2016 19:05:40 -0000 (UTC)
Subject: Guido? Where are you?
References: <o0s4l0$c8a$3@dont-email.me> <1853932.OBFZWjSADL@PointedEars.de>
 <e9fpjfFl65tU1@mid.individual.net> <5832B60D.906@stoneleaf.us>
 <mailman.16.1479718419.3146.python-list@python.org>
 <e20Zz.91860$7c1.34407@fx03.am4>
Message-ID: <slrno395p7.2bv.jon+usenet@sable.unequivocal.eu>

On 2016-11-22, Gilmeh Serda <gilmeh.serdah at nothing.here.invalid> wrote:
> On Mon, 21 Nov 2016 00:53:33 -0800, Ethan Furman wrote:
>> Unfortunately, we do not have any control over the comp.lang.python
>> newsgroup
>
> Gee, "unfortunately"? Really!? Gosh! I'm glad I don't have to live 
> anywhere close to you. 8?[
>
> NOBODY "owns" the groups, and rightfully so. If you want the groups to be 
> private, run your own server!

... or make them moderated, which I think they should do to this group,
so that the list mirroring works properly.


From bc at freeuk.com  Tue Nov 22 14:18:07 2016
From: bc at freeuk.com (BartC)
Date: Tue, 22 Nov 2016 19:18:07 +0000
Subject: Numpy slow at vector cross product?
In-Reply-To: <583476fa$0$1583$c3e8da3$5496439d@news.astraweb.com>
References: <o0t22o$d9r$3@dont-email.me>
 <58326062$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <o0uq77$cjf$1@blaine.gmane.org>
 <mailman.18.1479732277.3146.python-list@python.org>
 <o0v1oe$jju$1@dont-email.me> <pan.2016.11.21.17.04.32.390000@nowhere.invalid>
 <o0vf84$ble$1@dont-email.me>
 <5833b121$0$1603$c3e8da3$5496439d@news.astraweb.com>
 <o11b6o$rfr$1@dont-email.me>
 <CANc-5UzR27T4fkP_JodSNkp32wz1dwnePMHnjGdtwvffdHhd8w@mail.gmail.com>
 <mailman.34.1479818099.3146.python-list@python.org>
 <o11ejn$7i2$1@dont-email.me>
 <583476fa$0$1583$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o125kf$11v$1@dont-email.me>

On 22/11/2016 16:48, Steve D'Aprano wrote:
> On Tue, 22 Nov 2016 11:45 pm, BartC wrote:
>
>
>> I will have a look. Don't forget however that all someone is trying to
>> do is to multiply two vectors. They're not interested in axes
>> transformation or making them broadcastable, whatever that means.
>
> You don't know that.

When I did vector arithmetic at school, the formula for the 
cross-product of (x1,y1,z1) with (x2,y2,z2) was quite straightforward. 
This is what you expect given a library function to do the job. 
Especially in an implementation that could be crippled by running as 
unoptimised byte-code.

If someone was given the task of cross-multiplying two three-element 
vectors, the formula in the text-book (or Wikipedia) is what they would 
write.

> Could numpy optimize the single pair of vectors case a bit better? Perhaps
> they could. It looks like there's a bunch of minor improvements which could
> be made to the code, a few micro-optimizations that shave a microsecond or
> two off the execution time. And maybe they could even detect the case where
> the arguments are a single pair of vectors, and optimize that. Even
> replacing it with a naive pure-Python cross product would be a big win.

The code it ends up executing for individual pairs of vectors is a joke. 
97% of what it does is nothing to do with the cross-product!

> But for the big array of vectors case, you absolutely have to support doing
> fast vectorized cross-products over a huge number of vectors.

That's how I expected numpy to work. I said something along those lines 
in my first post:

BC:
 > Maybe numpy has extra overheads, and the arrays being operated on are
 > very small, but even so, 30 times slower than CPython? (2.5 to 0.083
 > seconds.)


> py> a = np.array([
> ...     [1, 2, 3],
> ...     [4, 5, 6],
> ...     [7, 8, 9],
> ...     ]*1000
> ...     )
> py> b = np.array([
> ...     [9, 8, 7],
> ...     [6, 5, 4],
> ...     [3, 2, 1],
> ...     ]*1000
> ...     )
> py> a.shape
> (3000, 3)
> py> result = np.cross(a, b)
> py> result.shape
> (3000, 3)
>
> On my computer, numpy took only 10 times longer to cross-multiply 3000 pairs
> of vectors than it took to cross-multiply a single pair of vectors. If I
> did that in pure Python, it would take 3000 times longer, or more, so numpy
> wins here by a factor of 300.

I tested this with 3 million pairs on my machine. It managed 8 million 
products per second (about the same as the simplest Python code, but run 
with pypy), taking nearly 0.4 seconds. Except it took over 4 seconds to 
create the special numpy arrays!

Setting up ordinary arrays was much faster, but then the cross-product 
calculation was slower.

-- 
bartc


From steven.truppe at chello.at  Tue Nov 22 15:33:30 2016
From: steven.truppe at chello.at (Steven Truppe)
Date: Tue, 22 Nov 2016 21:33:30 +0100
Subject: Question about working with html entities in python 2 to use them as
 filenames
Message-ID: <f662e584-c49f-1249-acce-2306a384e655@chello.at>

I all,


i'm using linux and python 2 and want to parse a file line by line by 
executing a command with the line (with os.system).

My problem now is that i'm opening the file and parse the title but i'm 
not able to get it into a normal filename:


import os,sys

import urlib,re,cgi

import HTMLParser, uincodedata

import htmlentiytdefs

imort chardet

for ULR in open('list.txt', "r").readlines():

     teste_egex="<title>(.+?)</title>

     patter = re.compile(these_regex)

     htmlfile=urlib.urlopen(URL)

     htmltext=htmlfile.read()

     title=re.aindall(pater, htmltext)[0]

     title = HTMLParser.HTMLParser.unescape(title)

     print "title = ", title

# here i would like to create a directory named after the content of the title


I allways get this error:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2



i've played around with .ecode('latin-1') or ('utf8') but i was not yet 
able to sove this simple issue.


Tanks in advance,

Truppe Steven



From steven.truppe at chello.at  Tue Nov 22 15:54:27 2016
From: steven.truppe at chello.at (Steven Truppe)
Date: Tue, 22 Nov 2016 21:54:27 +0100
Subject: Question about working with html entities in python 2 to use them
 as filenames
In-Reply-To: <f662e584-c49f-1249-acce-2306a384e655@chello.at>
References: <f662e584-c49f-1249-acce-2306a384e655@chello.at>
Message-ID: <bb91d053-8505-f235-067e-c5e895dcdb98@chello.at>

I've made a pastebin with a few examples: http://pastebin.com/QQQFhkRg



On 2016-11-22 21:33, Steven Truppe wrote:
> I all,
>
>
> i'm using linux and python 2 and want to parse a file line by line by 
> executing a command with the line (with os.system).
>
> My problem now is that i'm opening the file and parse the title but 
> i'm not able to get it into a normal filename:
>
>
> import os,sys
>
> import urlib,re,cgi
>
> import HTMLParser, uincodedata
>
> import htmlentiytdefs
>
> imort chardet
>
> for ULR in open('list.txt', "r").readlines():
>
>     teste_egex="<title>(.+?)</title>
>
>     patter = re.compile(these_regex)
>
>     htmlfile=urlib.urlopen(URL)
>
>     htmltext=htmlfile.read()
>
>     title=re.aindall(pater, htmltext)[0]
>
>     title = HTMLParser.HTMLParser.unescape(title)
>
>     print "title = ", title
>
> # here i would like to create a directory named after the content of 
> the title
>
>
> I allways get this error:
>
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2
>
>
>
> i've played around with .ecode('latin-1') or ('utf8') but i was not 
> yet able to sove this simple issue.
>
>
> Tanks in advance,
>
> Truppe Steven
>



From amirouche.boubekki at gmail.com  Tue Nov 22 15:56:30 2016
From: amirouche.boubekki at gmail.com (Amirouche Boubekki)
Date: Tue, 22 Nov 2016 20:56:30 +0000
Subject: Anyone needs a graphdb written in Python?
Message-ID: <CAL7_Mo_ANsYGv=qcDgwxEM4hKArPTQj68C2z1SJk33O4YJig9g@mail.gmail.com>

H?llo,


I am working on a graphdb written Python on top of wiredtiger.

Anyone want to share about the subject about where this could be made
useful?

TIA!


From martin.pyka at gmx.de  Tue Nov 22 16:18:38 2016
From: martin.pyka at gmx.de (Martin Pyka)
Date: Tue, 22 Nov 2016 22:18:38 +0100
Subject: new python package: financial_life
Message-ID: <5834B62E.8060908@gmx.de>

Hi folks,

for all data scientists and especially financial analysts out there, 
this python package might be a useful resource:

https://github.com/MartinPyka/financial_life

With financial_life, monetary flows between different bank accounts can 
be simulated with a few lines of code. These simulations can help to get 
a deeper understanding of financial plans and a better comparison of 
financial products (in particular loan conditions) for personal 
circumstances.

You can
- analyse loan conditions and payment strategies
- create dynamic monetary flows between accounts for modeling more 
realistic scenarios
- extend the code by controller functions (e.g. for modeling tax payments)

I wrote it to analyse my own financial plans. Maybe, this package is 
also helpful for some of you.

Best,
Martin




From lew.pitcher at digitalfreehold.ca  Tue Nov 22 17:00:47 2016
From: lew.pitcher at digitalfreehold.ca (Lew Pitcher)
Date: Tue, 22 Nov 2016 17:00:47 -0500
Subject: Question about working with html entities in python 2 to use them
 as filenames
References: <f662e584-c49f-1249-acce-2306a384e655@chello.at>
 <bb91d053-8505-f235-067e-c5e895dcdb98@chello.at>
 <mailman.39.1479848070.3146.python-list@python.org>
Message-ID: <o12f5d$64h$1@dont-email.me>

On Tuesday November 22 2016 15:54, in comp.lang.python, "Steven Truppe"
<steven.truppe at chello.at> wrote:

> I've made a pastebin with a few examples: http://pastebin.com/QQQFhkRg
> 
> 
> 
> On 2016-11-22 21:33, Steven Truppe wrote:
>> I all,
>>
>>
>> i'm using linux and python 2 and want to parse a file line by line by
>> executing a command with the line (with os.system).
>>
>> My problem now is that i'm opening the file and parse the title but
>> i'm not able to get it into a normal filename:
>>
>>
>> import os,sys
>>
>> import urlib,re,cgi
>>
>> import HTMLParser, uincodedata
>>
>> import htmlentiytdefs
>>
>> imort chardet
>>
>> for ULR in open('list.txt', "r").readlines():
>>
>>     teste_egex="<title>(.+?)</title>
>>
>>     patter = re.compile(these_regex)
>>
>>     htmlfile=urlib.urlopen(URL)
>>
>>     htmltext=htmlfile.read()
>>
>>     title=re.aindall(pater, htmltext)[0]
>>
>>     title = HTMLParser.HTMLParser.unescape(title)
>>
>>     print "title = ", title
>>
>> # here i would like to create a directory named after the content of
>> the title
>>
>>
>> I allways get this error:
>>
>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2
>>
>>
>>
>> i've played around with .ecode('latin-1') or ('utf8') but i was not
>> yet able to sove this simple issue.

I'm no python programmer, but I do have a couple of observations.

First, though, here's an extract from that pastebin posting of yours:
>     print "Title = ", title.decode()
>  
> ----- RESULT --------
> Title =  Wizo - Anderster Full Album - YouTube
> Title =  Wizo - Bleib Tapfer / f?r&#39;n Arsch Full Album - YouTube
> Title =  WIZO - Uuaarrgh Full Album - YouTube
> Title =  WIZO - Full Album - &quot;Punk gibt&#39;s nicht umsonst! (Teill
III)&quot; - YouTube
> Title =  WIZO - Full Album - &quot;DER&quot; - YouTube
> Title =  Alarmsignal -  Wir leben - YouTube
> Title =  the Pogues - Body of an american - YouTube
> Title =  The Pogues -  The band played waltzing matilda - YouTube
> Title =  Hey Rote Zora - Heiter bis Wolkig - YouTube
> Title =  F?r immer Punk - die goldenen Zitronen - YouTube
> Title =  Fuckin&#39; Faces - Krieg und Frieden - YouTube
> Title =  Sluts - Anders - YouTube
> Title =  Absturz - Es ist sch?n ein Punk zu sein - YouTube
> Title =  Broilers - Ruby Light &amp; Dark - YouTube
> Title =  Less Than Jake 02 - My Very Own Flag - YouTube
> Title =  The Mighty Mighty Bosstones - The Impression That I Get - YouTube
> Title =  Streetlight Manifesto - Failing Flailing (lyrics) - YouTube
> Title =  Mustard Plug - Mr. Smiley - YouTube
>  
> But when i try:
> os.mkdir(title)
> i get the following:
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 23:
> ordinal not in range(128) 

Now for the observations

1) some of your titles contain the '/' character, which on your platform
(Linux) is taken as a path separator character. The os.mkdir() method
apparently expects it's "path" argument to name a file in an already existing
directory. That is to say, if path is "/a/b/c", then os.mkdir() expects that
the directory /a/b will already exist. Those titles that contain the path
separator character will cause os.mkdir() to attempt to create a file in a
subdirectory of the current directory, and that subdirectory doesn't exist
yet. You either have to sanitize your input to remove the path separators,
and use os.mkdir() to create a file named with the name of the sanitized
path, /or/ use os.makedirs(), which will create all the subdirectories
required by your given path.

2) Apparently os.mkdir() (at least) defaults to requiring an ASCII pathname.
Those of your titles that contain Unicode characters cannot be stored
verbatim without either
  a) re-encoding the title in ASCII, or
  b) flagging to os.mkdir() that Unicode is acceptable.
Apparently, this is a common problem; a google search brought up several pages
dedicated to answering this question, including one extensive paper on the
topic (http://nedbatchelder.com/text/unipain.html). There apparently are ways
to cause os.mkdir() to accept Unicode inputs; their effectiveness and
side-effects are beyond me.

HTH
-- 
Lew Pitcher
"In Skills, We Trust"
PGP public key available upon request



From steve+python at pearwood.info  Tue Nov 22 19:27:44 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Wed, 23 Nov 2016 11:27:44 +1100
Subject: Question about working with html entities in python 2 to use them
 as filenames
References: <f662e584-c49f-1249-acce-2306a384e655@chello.at>
 <mailman.38.1479847507.3146.python-list@python.org>
Message-ID: <5834e282$0$1596$c3e8da3$5496439d@news.astraweb.com>

On Wed, 23 Nov 2016 07:33 am, Steven Truppe wrote:

> imort chardet

That's not working Python code.

Steven, you have asked us to help you with some code. For us to do that, we
need to see the ACTUAL code you are running, not some other code which is
full of typos and may be very different from what is actually being run.

Don't re-type your program, copy and paste it. And make sure that it is the
code that does what you say it does. We're volunteers, and it isn't very
nice to have us waste our time trying to fix your code only for you to then
later say "oh sorry, that was the wrong code".


> # here i would like to create a directory named after the content of the
> # title

This appears to be the critical code: you're saying you would like to create
a directory, but don't show us the code that creates the directory!

We're very clever, but we cannot read your mind.


> I allways get this error:
> 
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2

Are we supposed to guess where you get that error? Python gives you lots of
excellent debugging information: the traceback. The traceback shows you
which line of code fails with that error, and the full list of lines of
code calling it.

Please COPY and PASTE (don't re-type, don't summarise, don't simplify, and
especially don't take a screen shot) the entire traceback, starting from
the line beginning "Traceback" and going to the final error message.





-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Tue Nov 22 20:15:15 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Wed, 23 Nov 2016 12:15:15 +1100
Subject: Question about working with html entities in python 2 to use them
 as filenames
References: <f662e584-c49f-1249-acce-2306a384e655@chello.at>
 <bb91d053-8505-f235-067e-c5e895dcdb98@chello.at>
 <mailman.39.1479848070.3146.python-list@python.org>
Message-ID: <5834eda6$0$1622$c3e8da3$5496439d@news.astraweb.com>

On Wed, 23 Nov 2016 07:54 am, Steven Truppe wrote:

> I've made a pastebin with a few examples: http://pastebin.com/QQQFhkRg

Your pastebin appears to be the same code as you've shown here. And, again,
it doesn't seem to be the actual code you are really running.

The only new or helpful information is that you're trying to call

os.mkdir(title)

where title is, well, we have to guess, because you don't tell us.

My *guess* is that the failure happens when processing the title:

"Wizo - Bleib Tapfer / f?r&#39;n Arsch Full Album - YouTube"

but since we really don't know the actual code you are using, we have to
guess.

My guess is that you have the byte string:

title = "Wizo - Bleib Tapfer / f\xc3\xbcr&#39;n Arsch Full Album - YouTube"

Notice the \xc3 byte? You can check this by printing the repr() of the
title:

py> print repr(title)
'Wizo - Bleib Tapfer / f\xc3\xbcr&#39;n Arsch Full Album - YouTube'



When you try title.decode(), it fails:

py> print title.decode()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 23:
ordinal not in range(128)



But if you tell it to use UTF-8, it succeeds:

py> print title.decode('utf-8')
Wizo - Bleib Tapfer / f?r&#39;n Arsch Full Album - YouTube



So my guess is that you're tying to create the directory like this:


os.mkdir(title.decode())


and getting the same error. You should try:

os.mkdir(title.decode('utf-8'))

which will at least give you a new error: you cannot use '/' inside a
directory name. So you can start by doing this:


os.mkdir(title.replace('/', '-').decode('utf-8'))


and see what happens.


Beyond that, I cannot guess what you need to do to fix the code I haven't
seen.



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Tue Nov 22 20:32:52 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Wed, 23 Nov 2016 12:32:52 +1100
Subject: Question about working with html entities in python 2 to use them
 as filenames
References: <f662e584-c49f-1249-acce-2306a384e655@chello.at>
 <bb91d053-8505-f235-067e-c5e895dcdb98@chello.at>
 <mailman.39.1479848070.3146.python-list@python.org>
 <o12f5d$64h$1@dont-email.me>
Message-ID: <5834f1c7$0$22141$c3e8da3$5496439d@news.astraweb.com>

On Wed, 23 Nov 2016 09:00 am, Lew Pitcher wrote:

> 2) Apparently os.mkdir() (at least) defaults to requiring an ASCII
> pathname. 

No, you have misinterpreted what you have seen.

Even in Python 2, os.mkdir will accept a Unicode argument. You just have to
make sure it is given as unicode:

os.mkdir(u'/tmp/f?r')

Notice the u' delimiter instead of the ordinary ' delimiter? That tells
Python to use a unicode (text) string instead of an ascii byte-string.

If you don't remember the u' delimiter, and write an ordinary byte-string '
delimiter, then the result you get will depend on some combination of your
operating system, the source code encoding, and Python's best guess of what
you mean.

os.mkdir('/tmp/f?r')  # don't do this!

*might* work, if all the factors align correctly, but often won't. And when
it doesn't, the failure can be extremely mysterious, usually involving a
spurious 

UnicodeDecodeError: 'ascii' codec

error.

Dealing with Unicode text is much simpler in Python 3. Dealing with
*unknown* encodings is never easy, but so long as you can stick with
Unicode and UTF-8, Python 3 makes it easy. 




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From nathan.ernst at gmail.com  Tue Nov 22 21:50:40 2016
From: nathan.ernst at gmail.com (Nathan Ernst)
Date: Tue, 22 Nov 2016 20:50:40 -0600
Subject: Unexpected PendingDeprecationWarning
Message-ID: <CAPjBxg9a1WSpB1Vv6fvtez42etybH-PxHLLMmqQppzxq28cUtg@mail.gmail.com>

I'm using Python 3.5.2, and the following code (when invoked) causes a
PendingDeprecationWarning when used in a unit test:

def identity(x):
  return x

def adjacent_difference(seq, selector=identity):
  i = iter(seq)
  l = selector(next(i))
  while True:
    r = selector(next(i))
    yield r - l
    l = r

I wrote this to mimic the C++ std algorithm (defined here:
http://en.cppreference.com/w/cpp/algorithm/adjacent_difference).

What I don't understand is why I get this warning.

The exact error message I get from unittest is:
PendingDeprecationWarning: generator 'adjacent_difference' raised
StopIteration

I'd appreciate any insight into what is causing this deprecation warning,
as I am stumped.

Regards,
Nate


From python at mrabarnett.plus.com  Tue Nov 22 22:02:22 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Wed, 23 Nov 2016 03:02:22 +0000
Subject: Unexpected PendingDeprecationWarning
In-Reply-To: <CAPjBxg9a1WSpB1Vv6fvtez42etybH-PxHLLMmqQppzxq28cUtg@mail.gmail.com>
References: <CAPjBxg9a1WSpB1Vv6fvtez42etybH-PxHLLMmqQppzxq28cUtg@mail.gmail.com>
Message-ID: <2f7946d5-d8e7-7bae-b69d-020e821c43e4@mrabarnett.plus.com>

On 2016-11-23 02:50, Nathan Ernst wrote:
> I'm using Python 3.5.2, and the following code (when invoked) causes a
> PendingDeprecationWarning when used in a unit test:
>
> def identity(x):
>   return x
>
> def adjacent_difference(seq, selector=identity):
>   i = iter(seq)
>   l = selector(next(i))
>   while True:
>     r = selector(next(i))
>     yield r - l
>     l = r
>
> I wrote this to mimic the C++ std algorithm (defined here:
> http://en.cppreference.com/w/cpp/algorithm/adjacent_difference).
>
> What I don't understand is why I get this warning.
>
> The exact error message I get from unittest is:
> PendingDeprecationWarning: generator 'adjacent_difference' raised
> StopIteration
>
> I'd appreciate any insight into what is causing this deprecation warning,
> as I am stumped.
>
The 'while' loop keeps calling next(i) until it raises StopIteration, 
and that kind of behaviour can hide obscure bugs.

If you want to know the details, you can read the rationale behind the 
change in the relevant PEP:

PEP 479 -- Change StopIteration handling inside generators
https://www.python.org/dev/peps/pep-0479/



From nathan.ernst at gmail.com  Tue Nov 22 22:14:08 2016
From: nathan.ernst at gmail.com (Nathan Ernst)
Date: Tue, 22 Nov 2016 21:14:08 -0600
Subject: Unexpected PendingDeprecationWarning
In-Reply-To: <2f7946d5-d8e7-7bae-b69d-020e821c43e4@mrabarnett.plus.com>
References: <CAPjBxg9a1WSpB1Vv6fvtez42etybH-PxHLLMmqQppzxq28cUtg@mail.gmail.com>
 <2f7946d5-d8e7-7bae-b69d-020e821c43e4@mrabarnett.plus.com>
Message-ID: <CAPjBxg-zfzwFToQoMQ-49FPkULZQay098-KbXjtXzfh2L5DXsw@mail.gmail.com>

Thanks,

I was not aware of that PEP.

The logic in my function is exactly as desired, so to squelch the warning,
I merely wrapped the iteration in a try/except:

def adjacent_difference(seq, selector=identity):
  i = iter(seq)
  l = selector(next(i))
  try:
    while True:
      r = selector(next(i))
      yield r - l
      l = r
  except StopIteration:
    return


On Tue, Nov 22, 2016 at 9:02 PM, MRAB <python at mrabarnett.plus.com> wrote:

> On 2016-11-23 02:50, Nathan Ernst wrote:
>
>> I'm using Python 3.5.2, and the following code (when invoked) causes a
>> PendingDeprecationWarning when used in a unit test:
>>
>> def identity(x):
>>   return x
>>
>> def adjacent_difference(seq, selector=identity):
>>   i = iter(seq)
>>   l = selector(next(i))
>>   while True:
>>     r = selector(next(i))
>>     yield r - l
>>     l = r
>>
>> I wrote this to mimic the C++ std algorithm (defined here:
>> http://en.cppreference.com/w/cpp/algorithm/adjacent_difference).
>>
>> What I don't understand is why I get this warning.
>>
>> The exact error message I get from unittest is:
>> PendingDeprecationWarning: generator 'adjacent_difference' raised
>> StopIteration
>>
>> I'd appreciate any insight into what is causing this deprecation warning,
>> as I am stumped.
>>
>> The 'while' loop keeps calling next(i) until it raises StopIteration, and
> that kind of behaviour can hide obscure bugs.
>
> If you want to know the details, you can read the rationale behind the
> change in the relevant PEP:
>
> PEP 479 -- Change StopIteration handling inside generators
> https://www.python.org/dev/peps/pep-0479/
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From rosuav at gmail.com  Tue Nov 22 22:23:52 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 23 Nov 2016 14:23:52 +1100
Subject: Unexpected PendingDeprecationWarning
In-Reply-To: <CAPjBxg9a1WSpB1Vv6fvtez42etybH-PxHLLMmqQppzxq28cUtg@mail.gmail.com>
References: <CAPjBxg9a1WSpB1Vv6fvtez42etybH-PxHLLMmqQppzxq28cUtg@mail.gmail.com>
Message-ID: <CAPTjJmqyQtcitsU5xw182GN3JMOZnF_vCV7CQOy9s74pw9bmSg@mail.gmail.com>

On Wed, Nov 23, 2016 at 1:50 PM, Nathan Ernst <nathan.ernst at gmail.com> wrote:
> I'm using Python 3.5.2, and the following code (when invoked) causes a
> PendingDeprecationWarning when used in a unit test:
>
> def identity(x):
>   return x
>
> def adjacent_difference(seq, selector=identity):
>   i = iter(seq)
>   l = selector(next(i))
>   while True:
>     r = selector(next(i))
>     yield r - l
>     l = r
>
> I wrote this to mimic the C++ std algorithm (defined here:
> http://en.cppreference.com/w/cpp/algorithm/adjacent_difference).
>
> What I don't understand is why I get this warning.
>
> The exact error message I get from unittest is:
> PendingDeprecationWarning: generator 'adjacent_difference' raised
> StopIteration
>
> I'd appreciate any insight into what is causing this deprecation warning,
> as I am stumped.

It's because there are some extremely confusing possibilities when a
generator raises StopIteration. In the example you post above, you're
probably expecting the behaviour you do indeed get, but there are
other ways of writing generators that would be a lot more surprising,
particularly when you refactor the generator a bit. The future
behaviour is much simpler to explain: *any* exception is an error, and
the way to cleanly terminate the generator is to return from it.

Here's how you'd write that generator for a post-3.6 world:

def adjacent_difference(seq, selector=identity):
  i = iter(seq)
  try: l = selector(next(i))
  except StopIteration: return
  for r in i:
    r = selector(r)
    yield r - l
    l = r

This is more explicit about the behaviour in the face of an empty
iterable (it will return without yielding any results), and uses the
most obvious way of stepping an iterator, namely a 'for' loop. You're
now iterating through 'i' and yielding stuff.

As a general rule, you should be able to replace 'yield x' with
'print(x)' and the code will do the same thing, only printing to the
console instead of being iterated over. Deliberately allowing
StopIteration to leak breaks that.

Here's the doc that set it all out:

https://www.python.org/dev/peps/pep-0479/

ChrisA


From rosuav at gmail.com  Tue Nov 22 22:24:46 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 23 Nov 2016 14:24:46 +1100
Subject: Unexpected PendingDeprecationWarning
In-Reply-To: <CAPjBxg-zfzwFToQoMQ-49FPkULZQay098-KbXjtXzfh2L5DXsw@mail.gmail.com>
References: <CAPjBxg9a1WSpB1Vv6fvtez42etybH-PxHLLMmqQppzxq28cUtg@mail.gmail.com>
 <2f7946d5-d8e7-7bae-b69d-020e821c43e4@mrabarnett.plus.com>
 <CAPjBxg-zfzwFToQoMQ-49FPkULZQay098-KbXjtXzfh2L5DXsw@mail.gmail.com>
Message-ID: <CAPTjJmqsSfkOC702RVut_Yf+G5Z0rNzjT2Tv_F3jDDbk6Hk7og@mail.gmail.com>

On Wed, Nov 23, 2016 at 2:14 PM, Nathan Ernst <nathan.ernst at gmail.com> wrote:
> I was not aware of that PEP.
>
> The logic in my function is exactly as desired, so to squelch the warning,
> I merely wrapped the iteration in a try/except:
>
> def adjacent_difference(seq, selector=identity):
>   i = iter(seq)
>   l = selector(next(i))
>   try:
>     while True:
>       r = selector(next(i))
>       yield r - l
>       l = r
>   except StopIteration:
>     return

You'll probably want to move the 'try' up one line - unless you want
an exception if the sequence is empty. And it's exactly this kind of
ambiguity that this change helps to catch.

ChrisA


From nathan.ernst at gmail.com  Tue Nov 22 23:03:12 2016
From: nathan.ernst at gmail.com (Nathan Ernst)
Date: Tue, 22 Nov 2016 22:03:12 -0600
Subject: Unexpected PendingDeprecationWarning
In-Reply-To: <CAPTjJmqsSfkOC702RVut_Yf+G5Z0rNzjT2Tv_F3jDDbk6Hk7og@mail.gmail.com>
References: <CAPjBxg9a1WSpB1Vv6fvtez42etybH-PxHLLMmqQppzxq28cUtg@mail.gmail.com>
 <2f7946d5-d8e7-7bae-b69d-020e821c43e4@mrabarnett.plus.com>
 <CAPjBxg-zfzwFToQoMQ-49FPkULZQay098-KbXjtXzfh2L5DXsw@mail.gmail.com>
 <CAPTjJmqsSfkOC702RVut_Yf+G5Z0rNzjT2Tv_F3jDDbk6Hk7og@mail.gmail.com>
Message-ID: <CAPjBxg-tG9oTg0H-iuSMAxBgoVc2FR-Xp6RYJfmyY7e=QKrgbA@mail.gmail.com>

Thanks, ChrisA

On Tue, Nov 22, 2016 at 9:24 PM, Chris Angelico <rosuav at gmail.com> wrote:

> On Wed, Nov 23, 2016 at 2:14 PM, Nathan Ernst <nathan.ernst at gmail.com>
> wrote:
> > I was not aware of that PEP.
> >
> > The logic in my function is exactly as desired, so to squelch the
> warning,
> > I merely wrapped the iteration in a try/except:
> >
> > def adjacent_difference(seq, selector=identity):
> >   i = iter(seq)
> >   l = selector(next(i))
> >   try:
> >     while True:
> >       r = selector(next(i))
> >       yield r - l
> >       l = r
> >   except StopIteration:
> >     return
>
> You'll probably want to move the 'try' up one line - unless you want
> an exception if the sequence is empty. And it's exactly this kind of
> ambiguity that this change helps to catch.
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From no.email at nospam.invalid  Wed Nov 23 00:43:46 2016
From: no.email at nospam.invalid (Paul Rubin)
Date: Tue, 22 Nov 2016 21:43:46 -0800
Subject: Question about working with html entities in python 2 to use them
 as filenames
References: <f662e584-c49f-1249-acce-2306a384e655@chello.at>
 <mailman.38.1479847507.3146.python-list@python.org>
Message-ID: <87eg22agz1.fsf@nightsong.com>

Steven Truppe <steven.truppe at chello.at> writes:

> # here i would like to create a directory named after the content of
> # the title... I allways get this error:
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2

The title has a ? (capital A with tilde) character in it, and there is
no corresponding ascii character.  So you can't encode that string into
ascii.  You can encode it into utf8 or whatever, but are you on an OS
that recognizes utf8 in filenames?  Maybe you want to transcode it
somehow.  Otherwise you may be asking for trouble if some of those html
strings have control characters and stuff in them.

Also, if you're scraping web pages, you may have an easier time with
BeautifulSoup (search web for it) than HTMLparser.


From orgnut at yahoo.com  Wed Nov 23 03:20:23 2016
From: orgnut at yahoo.com (Larry Hudson)
Date: Wed, 23 Nov 2016 00:20:23 -0800
Subject: Can somebody tell me what's wrong wrong with my code? I don't
 understand
In-Reply-To: <e08ef692-5f20-494b-bb3e-676a6423ccf6@googlegroups.com>
References: <e08ef692-5f20-494b-bb3e-676a6423ccf6@googlegroups.com>
Message-ID: <-LqdnVWVqaHVzKjFnZ2dnUU7-TnNnZ2d@giganews.com>

On 11/21/2016 07:10 PM, rmjbros3 at gmail.com wrote:
> Hi! This is my first post! I'm having trouble understanding my code. I get "SyntaxError:invalid syntax" on line 49. I'm trying to code a simple text-based rpg on repl.it. Thank you for reading.
>
>
>
> print("Welcome to Gladiator Game! Choose your character race, class, and starting equipment!")
>
> print('')
>
> print("Race selection: ")
>
> print('')
>
> print("(1) Orcs. Known for their very wide, robust physiques. They are the strongest of all the races in Polaris.")
>
> print('')
>
> print("(2) Elves. Thin and wiry. They are known for their amazing agility and hand-eye coordiation. They originate from the desert island of Angolia.")
>
> print('')
>
> print("(3) Silverbacks. A hairy, ape-like race from Nothern Polaris. Their metal fur provides them with much needed protection.")
>
> print('')
>
> print("(4) Pomongos. An amphibian race believed to inhabit the wet jungles of Central Polaris. Legends say they have highly corrosive spit...")
>
> print('')
>
> raceNum=int(input("Select your character's race by entering the corresponding number. Then press enter: "))
>
> print('')
>
> while raceNum<1 or raceNum>4:
>   raceNum=int(input('Invalid input. Try again: '))
>
> print('')
>
> if raceNum==1:
>   print("You're an orc, eh? I won't be sayin' anything mean about you...")
>   print('')
>   classNum=int(input("What's your profession big fella?"))
>
> elif raceNum==2:
>   print("I never liked you elven folk...Let's get on with this.")
>   print('')
>   classNum=int(input("What's your profession ? Do ye even have one ?"))
>
> elif raceNum==3:
>   print("Nice fur. I don't see too many of your kind 'round here. Maybe that's a good thing...")
>   print('')
>   classNum=int(input("What's your profession mate?")
>
> elif raceNum==4:             #this line has an error for some reason
>   print("Your a 'Mongo eh? I thought you lads were extinct...Just keep your tongue in ya mouth and we'll get along fine mate.")
>   classNum=int(input("What's your profession?"))
>

You've already received answers about your typo so I won't repeat that.  However I have a couple 
minor suggestions about your approach.

First, the empty string in your print('') statements is unnecessary.  Just use print(), it does 
the same thing.

Next your menu for the raceNum selection is straight-forward, and there is nothing wrong with 
that.  But another approach that might be a bit shorter is to use a list of strings displayed in 
a loop.  Something like this (very simplified) example...

<code>
options = ['First option',
     'Next option',
     'One more time',
     #  etc........
     ]
#  This version assumes the option number is part of the strings
for opt in options:
     print(opt)
     print()    #  If you really want the double-spacing, personally I don't think it's needed
</code>

Here's a bit more advanced version of the for loop, this one assumnes the option numbers are not 
in the strings in the list...

<code>
for num, opt in enumerate(options, 1):
     print('({})  {}'.format(num, opt))    #  OR   print('(%d)  %s' % (num, opt))
     print()    #  Again, double-spacing is optional
<code>

A similar approach could be used for your classNum section.

Just some suggestions to read/study/adapt...  or ignore.  Whatever you feel like.  ;-)

-- 
      -=- Larry -=-


From orgnut at yahoo.com  Wed Nov 23 03:42:47 2016
From: orgnut at yahoo.com (Larry Hudson)
Date: Wed, 23 Nov 2016 00:42:47 -0800
Subject: How to you convert list of tuples to string
In-Reply-To: <mailman.37.1479833528.3146.python-list@python.org>
References: <CACT3xuX0BM3txkDttjp8Nebn6DaDLUMyVQxuC0t3Mj27VHjc6w@mail.gmail.com>
 <CDC17C8A-463D-4247-B1BB-D1D46A6243E3@xs4all.nl>
 <mailman.37.1479833528.3146.python-list@python.org>
Message-ID: <Tu6dnbB_-s4Vy6jFnZ2dnUU7-eHNnZ2d@giganews.com>

On 11/22/2016 08:51 AM, Michiel Overtoom wrote:
> Hi Ganesh,
>
>> Any better suggestion to improve this piece of code and make it look more pythonic?
>
>
> import random
>
> # A list of tuples. Note that the L behind a number means that the number is a 'long'.
>
> data = [(1, 1, 373891072L, 8192), (1, 3, 390348800L, 8192), (1, 4, 372719616L,
>     8192), (2, 3, 382140416L, 8192), (2, 5, 398721024L, 8192), (3, 1,
>     374030336L, 8192), (3, 3, 374079488L, 8192), (3, 5, 340058112L, 8192)]
>
> item = random.choice(data)  # Select a random item from the 'data' list.
>
> msg = "%d,%d,%d:%d" % item  # Format it in the way you like.
>
> print msg
>
>
> Greetings,
>

Or using the new string formatting syntax:

msg = '{},{},{}:{}'.format(*item)

The *item in the format() unpacks the tuple.

-- 
      -=- Larry -=-


From frank at chagford.com  Wed Nov 23 04:10:42 2016
From: frank at chagford.com (Frank Millman)
Date: Wed, 23 Nov 2016 11:10:42 +0200
Subject: Is this pythonic?
Message-ID: <o13meh$p2g$1@blaine.gmane.org>

Hi all

Sometimes I write something that I think is quite clever, but later on I 
look at it and ask 'What was I thinking?'.

I have just come up with a 'clever' solution to a problem. Would this cause 
raised eyebrows if you were reviewing this?

I have a class that represents a single database column - there could be 
hundreds of instances at any time.

The class has a getval() method to return the current value.

Usually the value is stored in the instance, and can be returned 
immediately, but sometimes it has to be computed, incurring further database 
lookups.

In many cases the computed value is never actually requested, so I want to 
delay the computation until the first call to getval().

I could add an 'if computation_required: ' block to getval(), but I am 
trying to avoid that, partly because this would have to be checked for every 
call to getval() but would only used in a small number of cases, and partly 
because I have a few subclasses where getval() is over-ridden so I would 
have to add the extra code to every one (or call the superclass on every 
one).

This is what I have come up with.

1. Rename all instances of 'getval()' to '_getval()'.

2. Add a new method '_getval_with_comp()'.

3. When instantiating an object, check if it would need computation -
    if computation_required:
        self.getval = self._getval_with_comp
    else:
        self.getval = self._getval

4. In _getval_with_comp, perform the computation, then add the following -
        self.getval = self._getval
        return self._getval()

What is the verdict? -1, 0, or +1?

Frank Millman




From marko at pacujo.net  Wed Nov 23 04:50:14 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Wed, 23 Nov 2016 11:50:14 +0200
Subject: Is this pythonic?
References: <o13meh$p2g$1@blaine.gmane.org>
 <mailman.48.1479892259.3146.python-list@python.org>
Message-ID: <87inrer0dl.fsf@elektro.pacujo.net>

"Frank Millman" <frank at chagford.com>:

> 3. When instantiating an object, check if it would need computation -
>    if computation_required:
>        self.getval = self._getval_with_comp
>    else:
>        self.getval = self._getval
>
> 4. In _getval_with_comp, perform the computation, then add the following -
>        self.getval = self._getval
>        return self._getval()
>
> What is the verdict? -1, 0, or +1?

Perfectly cromulent, run-of-the-mill Python code.


Marko


From frank at chagford.com  Wed Nov 23 05:04:53 2016
From: frank at chagford.com (Frank Millman)
Date: Wed, 23 Nov 2016 12:04:53 +0200
Subject: Is this pythonic?
In-Reply-To: <87inrer0dl.fsf@elektro.pacujo.net>
References: <o13meh$p2g$1@blaine.gmane.org>
 <mailman.48.1479892259.3146.python-list@python.org>
 <87inrer0dl.fsf@elektro.pacujo.net>
Message-ID: <o13pk4$2j3$1@blaine.gmane.org>

"Marko Rauhamaa"  wrote in message news:87inrer0dl.fsf at elektro.pacujo.net...

> "Frank Millman" <frank at chagford.com>:

>
> > What is the verdict? -1, 0, or +1?
>
> Perfectly cromulent, run-of-the-mill Python code.
>

A new word to add to my vocabulary - thanks :-)

Frank




From steven.truppe at chello.at  Wed Nov 23 05:17:35 2016
From: steven.truppe at chello.at (Steven Truppe)
Date: Wed, 23 Nov 2016 11:17:35 +0100
Subject: Question about working with html entities in python 2 to use them
 as filenames
In-Reply-To: <5834f1c7$0$22141$c3e8da3$5496439d@news.astraweb.com>
References: <f662e584-c49f-1249-acce-2306a384e655@chello.at>
 <bb91d053-8505-f235-067e-c5e895dcdb98@chello.at>
 <mailman.39.1479848070.3146.python-list@python.org>
 <o12f5d$64h$1@dont-email.me>
 <5834f1c7$0$22141$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <9774898c-1d86-6529-bb21-0f9b40e2e9c5@chello.at>

type= <type 'str'> title =  Wizo - Anderster Full Album - YouTube
type= <type 'str'> title =  Wizo - Bleib Tapfer / f?r&#39;n Arsch Full 
Album - YouTube
Traceback (most recent call last):
   File "./music-fetcher.py", line 39, in <module>
     title = HTMLParser.HTMLParser().unescape(title)
   File "/usr/lib/python2.7/HTMLParser.py", line 475, in unescape
     return re.sub(r"&(#?[xX]?(?:[0-9a-fA-F]+|\w{1,8}));", 
replaceEntities, s)
   File "/usr/lib/python2.7/re.py", line 155, in sub
     return _compile(pattern, flags).sub(repl, string, count)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 23: 
ordinal not in range(128)

The pastebins from below are showing how i parse the html data from the 
<title> and i wan to have a normal filename with ?,? etc if possible,

i've tried converting with decode('utf-8') and encode or str.encode() 
and other thing but i think i'm missing here.


I want to create filename out of the <title>DATA</title>, it's realy 
important.


Hope in regards,

Truppe Steven


On 2016-11-23 02:32, Steve D'Aprano wrote:
> On Wed, 23 Nov 2016 09:00 am, Lew Pitcher wrote:
>
>> 2) Apparently os.mkdir() (at least) defaults to requiring an ASCII
>> pathname.
> No, you have misinterpreted what you have seen.
>
> Even in Python 2, os.mkdir will accept a Unicode argument. You just have to
> make sure it is given as unicode:
>
> os.mkdir(u'/tmp/f?r')
>
> Notice the u' delimiter instead of the ordinary ' delimiter? That tells
> Python to use a unicode (text) string instead of an ascii byte-string.
>
> If you don't remember the u' delimiter, and write an ordinary byte-string '
> delimiter, then the result you get will depend on some combination of your
> operating system, the source code encoding, and Python's best guess of what
> you mean.
>
> os.mkdir('/tmp/f?r')  # don't do this!
>
> *might* work, if all the factors align correctly, but often won't. And when
> it doesn't, the failure can be extremely mysterious, usually involving a
> spurious
>
> UnicodeDecodeError: 'ascii' codec
>
> error.
>
> Dealing with Unicode text is much simpler in Python 3. Dealing with
> *unknown* encodings is never easy, but so long as you can stick with
> Unicode and UTF-8, Python 3 makes it easy.
>
>
>
>



From ned at nedbatchelder.com  Wed Nov 23 06:09:58 2016
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Wed, 23 Nov 2016 03:09:58 -0800 (PST)
Subject: How to you convert list of tuples to string
In-Reply-To: <Tu6dnbB_-s4Vy6jFnZ2dnUU7-eHNnZ2d@giganews.com>
References: <CACT3xuX0BM3txkDttjp8Nebn6DaDLUMyVQxuC0t3Mj27VHjc6w@mail.gmail.com>
 <CDC17C8A-463D-4247-B1BB-D1D46A6243E3@xs4all.nl>
 <mailman.37.1479833528.3146.python-list@python.org>
 <Tu6dnbB_-s4Vy6jFnZ2dnUU7-eHNnZ2d@giganews.com>
Message-ID: <fc2236f6-783d-490f-afcf-830841ff0633@googlegroups.com>

On Wednesday, November 23, 2016 at 3:43:05 AM UTC-5, Larry Hudson wrote:
> On 11/22/2016 08:51 AM, Michiel Overtoom wrote:
> > Hi Ganesh,
> >
> >> Any better suggestion to improve this piece of code and make it look more pythonic?
> >
> >
> > import random
> >
> > # A list of tuples. Note that the L behind a number means that the number is a 'long'.
> >
> > data = [(1, 1, 373891072L, 8192), (1, 3, 390348800L, 8192), (1, 4, 372719616L,
> >     8192), (2, 3, 382140416L, 8192), (2, 5, 398721024L, 8192), (3, 1,
> >     374030336L, 8192), (3, 3, 374079488L, 8192), (3, 5, 340058112L, 8192)]
> >
> > item = random.choice(data)  # Select a random item from the 'data' list.
> >
> > msg = "%d,%d,%d:%d" % item  # Format it in the way you like.
> >
> > print msg
> >
> >
> > Greetings,
> >
> 
> Or using the new string formatting syntax:
> 
> msg = '{},{},{}:{}'.format(*item)
> 
> The *item in the format() unpacks the tuple.


"new" == "Introduced in 2.6, available since 2008" :)

--Ned.


From frank at chagford.com  Wed Nov 23 06:11:53 2016
From: frank at chagford.com (Frank Millman)
Date: Wed, 23 Nov 2016 13:11:53 +0200
Subject: Is this pythonic?
In-Reply-To: <87inrer0dl.fsf@elektro.pacujo.net>
References: <o13meh$p2g$1@blaine.gmane.org>
 <mailman.48.1479892259.3146.python-list@python.org>
 <87inrer0dl.fsf@elektro.pacujo.net>
Message-ID: <o13thn$b0m$1@blaine.gmane.org>

"Marko Rauhamaa"  wrote in message news:87inrer0dl.fsf at elektro.pacujo.net...

"Frank Millman" <frank at chagford.com>:
>
> > 3. When instantiating an object, check if it would need computation -
> >    if computation_required:
> >        self.getval = self._getval_with_comp
> >    else:
> >        self.getval = self._getval
> >
> > 4. In _getval_with_comp, perform the computation, then add the 
> > following -
> >        self.getval = self._getval
> >        return self._getval()
> >
> > What is the verdict? -1, 0, or +1?
>
> Perfectly cromulent, run-of-the-mill Python code.
>

Gah! The law of unintended consequences strikes again!

As I mentioned, the class in question represents a database column. A 
separate class represents a database row. I have a __str__() method on the 
'row' class that prints a nicely formatted representation of the object with 
all of its column objects and their values.

With the above changes, I had to turn getval() into a coroutine. My 
__str__() method uses getval() to obtain the values, so I had to prefix 
getval() with 'await', but then I get a syntax error on __str__(). I can add 
'async' to remove the syntax error, but then print(obj) does not work - 
TypeError: __str__ returned non-string (type coroutine)

I don't think there is an answer to this, but any suggestions will be 
appreciated.

I can say 'print(await obj.__str__())', and it works, but I lose the ability 
to include it in a larger print statement.

Ah well :-(

Frank




From steve+python at pearwood.info  Wed Nov 23 06:15:15 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Wed, 23 Nov 2016 22:15:15 +1100
Subject: Is this pythonic?
References: <o13meh$p2g$1@blaine.gmane.org>
 <mailman.48.1479892259.3146.python-list@python.org>
Message-ID: <58357a45$0$1589$c3e8da3$5496439d@news.astraweb.com>

On Wed, 23 Nov 2016 08:10 pm, Frank Millman wrote:

[...]
> The class has a getval() method to return the current value.
> 
> Usually the value is stored in the instance, and can be returned
> immediately, but sometimes it has to be computed, incurring further
> database lookups.

This is called memoisation, or caching, and is a perfectly standard
programming technique. It's not without its traps though: there's a famous
quote that says there are only two hard problems in computing, naming
things and cache invalidation. But putting that aside:


def getval(self):
    sentinel = object()
    value = getattr(self, '_cached_value', sentinel)
    if value is sentinel:
        # compute the value and store it
        value = ...
        self._cached_value = value
    return value

To invalidate the cache and force a recalculation:

    del self._cached_value


Now it's easy to override:

class Subclass(ParentClass):
    def getval(self):
        value = super(Subclass, self).getval()
        return value + 1



> This is what I have come up with.
> 
> 1. Rename all instances of 'getval()' to '_getval()'.
> 
> 2. Add a new method '_getval_with_comp()'.
> 
> 3. When instantiating an object, check if it would need computation -
>     if computation_required:
>         self.getval = self._getval_with_comp
>     else:
>         self.getval = self._getval

So this check only happens once, on instantiation? And you're sure that once
the instance is created, there will never be any circumstances where you
want to re-calculate the value?



> 4. In _getval_with_comp, perform the computation, then add the following -
>         self.getval = self._getval
>         return self._getval()

So you have something like this?

def _getval(self):
    return self._cached_value

def __getval_with_comp(self):
    value = ... # long computation
    self._cached_value = value
    self.getval = self._getval
    # return value
    return self._getval()
    # why call the method when you already know the answer?

    
How are subclasses supposed to override getval?




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From __peter__ at web.de  Wed Nov 23 06:21:14 2016
From: __peter__ at web.de (Peter Otten)
Date: Wed, 23 Nov 2016 12:21:14 +0100
Subject: Is this pythonic?
References: <o13meh$p2g$1@blaine.gmane.org>
Message-ID: <o13u36$k38$1@blaine.gmane.org>

Frank Millman wrote:

> Hi all
> 
> Sometimes I write something that I think is quite clever, but later on I
> look at it and ask 'What was I thinking?'.
> 
> I have just come up with a 'clever' solution to a problem. Would this
> cause raised eyebrows if you were reviewing this?
> 
> I have a class that represents a single database column - there could be
> hundreds of instances at any time.
> 
> The class has a getval() method to return the current value.
> 
> Usually the value is stored in the instance, and can be returned
> immediately, but sometimes it has to be computed, incurring further
> database lookups.
> 
> In many cases the computed value is never actually requested, so I want to
> delay the computation until the first call to getval().
> 
> I could add an 'if computation_required: ' block to getval(), but I am
> trying to avoid that, partly because this would have to be checked for
> every call to getval() but would only used in a small number of cases, and
> partly because I have a few subclasses where getval() is over-ridden so I
> would have to add the extra code to every one (or call the superclass on
> every one).
> 
> This is what I have come up with.
> 
> 1. Rename all instances of 'getval()' to '_getval()'.
> 
> 2. Add a new method '_getval_with_comp()'.
> 
> 3. When instantiating an object, check if it would need computation -
>     if computation_required:
>         self.getval = self._getval_with_comp
>     else:
>         self.getval = self._getval

You can also have the method replace itself:

>>> class Foo:
...     def get_val(self):
...         print("computing...")
...         val = self._val = 42
...         self.get_val = self.get_cached_val
...         return val
...     def get_cached_val(self):
...         return self._val
... 
>>> foo = Foo()
>>> foo.get_val()
computing...
42
>>> foo.get_val()
42

 
> 4. In _getval_with_comp, perform the computation, then add the following -
>         self.getval = self._getval
>         return self._getval()
> 
> What is the verdict? -1, 0, or +1?
> 
> Frank Millman




From rosuav at gmail.com  Wed Nov 23 06:22:53 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 23 Nov 2016 22:22:53 +1100
Subject: Is this pythonic?
In-Reply-To: <o13thn$b0m$1@blaine.gmane.org>
References: <o13meh$p2g$1@blaine.gmane.org>
 <mailman.48.1479892259.3146.python-list@python.org>
 <87inrer0dl.fsf@elektro.pacujo.net> <o13thn$b0m$1@blaine.gmane.org>
Message-ID: <CAPTjJmoZSoOR3vNbX9+JUPkEGxu17YZ7v1yHwAH51Rq50dYn8A@mail.gmail.com>

On Wed, Nov 23, 2016 at 10:11 PM, Frank Millman <frank at chagford.com> wrote:
> Gah! The law of unintended consequences strikes again!
>
> As I mentioned, the class in question represents a database column. A
> separate class represents a database row. I have a __str__() method on the
> 'row' class that prints a nicely formatted representation of the object with
> all of its column objects and their values.
>
> With the above changes, I had to turn getval() into a coroutine. My
> __str__() method uses getval() to obtain the values, so I had to prefix
> getval() with 'await', but then I get a syntax error on __str__(). I can add
> 'async' to remove the syntax error, but then print(obj) does not work -
> TypeError: __str__ returned non-string (type coroutine)
>
> I don't think there is an answer to this, but any suggestions will be
> appreciated.
>
> I can say 'print(await obj.__str__())', and it works, but I lose the ability
> to include it in a larger print statement.
>
> Ah well :-(

This strongly suggests that str(x) is the wrong way to get the
information. You shouldn't be doing database requests inside __str__
or __repr__.

ChrisA


From python.list at tim.thechases.com  Wed Nov 23 07:20:49 2016
From: python.list at tim.thechases.com (Tim Chase)
Date: Wed, 23 Nov 2016 06:20:49 -0600
Subject: Is this pythonic?
In-Reply-To: <58357a45$0$1589$c3e8da3$5496439d@news.astraweb.com>
References: <o13meh$p2g$1@blaine.gmane.org>
 <mailman.48.1479892259.3146.python-list@python.org>
 <58357a45$0$1589$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <20161123062049.07c260d2@bigbox.christie.dr>

On 2016-11-23 22:15, Steve D'Aprano wrote:
> On Wed, 23 Nov 2016 08:10 pm, Frank Millman wrote:
> > The class has a getval() method to return the current value.
> > 
> > Usually the value is stored in the instance, and can be returned
> > immediately, but sometimes it has to be computed, incurring
> > further database lookups.  
> 
> This is called memoisation, or caching, and is a perfectly standard
> programming technique. It's not without its traps though: there's a
> famous quote that says there are only two hard problems in
> computing, naming things and cache invalidation.

Fortunately, you can offload some odd edge-cases to the standard
library, no?

  from functools import lru_cache
  # ...
  @lru_cache(maxsize=1)
  def getval(...):
    return long_computation()

It doesn't cache across multiple instances of the same class, but
does cache multiple calls to the same instance's function:

  >>> from functools import lru_cache
  >>> class Foo:
  ...   def __init__(self, name):
  ...     self.name = name
  ...   @lru_cache(maxsize=1)
  ...   def getval(self):
  ...     print("Long process")
  ...     return self.name
  ... 
  >>> f1 = Foo("f1")
  >>> f2 = Foo("f2")
  >>> f1.getval()
  Long process
  'f1'
  >>> f1.getval()
  'f1'
  >>> f2.getval()
  Long process
  'f2'
  >>> f2.getval()
  'f2'

-tkc







From frank at chagford.com  Wed Nov 23 07:27:08 2016
From: frank at chagford.com (Frank Millman)
Date: Wed, 23 Nov 2016 14:27:08 +0200
Subject: Is this pythonic?
In-Reply-To: <o13meh$p2g$1@blaine.gmane.org>
References: <o13meh$p2g$1@blaine.gmane.org>
Message-ID: <o141us$il0$1@blaine.gmane.org>

"Frank Millman"  wrote in message news:o13meh$p2g$1 at blaine.gmane.org...

> 3. When instantiating an object, check if it would need computation -
>     if computation_required:
>         self.getval = self._getval_with_comp
>     else:
>         self.getval = self._getval
>
> 4. In _getval_with_comp, perform the computation, then add the following -
>        self.getval = self._getval
>         return self._getval()
>
> What is the verdict? -1, 0, or +1?
>

Thanks for the responses. I will reply to them all here -

@Peter
> You can also have the method replace itself ...

I like it. Thanks for the suggestion.

@Steve
> So this check only happens once, on instantiation? And you're sure that 
> once
> the instance is created, there will never be any circumstances where you
> want to re-calculate the value?

Well, the process that I call 'computation' includes setting up some 
variables that will trigger a recalculation when certain values change.

Part of my motivation was to avoid all of this if the value is never 
accessed.

> def __getval_with_comp(self):
>     value = ... # long computation
>     self._cached_value = value
>     self.getval = self._getval
>     # return value
>     return self._getval()
>     # why call the method when you already know the answer?
>
> How are subclasses supposed to override getval?
>

Two questions there, but the answer is the same.

I don't want subclasses to override the computation part of the process. I 
just want then to 'massage' the result before returning it. Therefore the 
answer to the first question is, to force the subclass to return the result, 
if it has its own _getval(). The answer to the second question is that they 
override _getval(), and therefore they will be invoked when getval() is 
called, provided getval has been set to be equal to _getval.

Hope that makes sense.

@Chris
> This strongly suggests that str(x) is the wrong way to get the
> information. You shouldn't be doing database requests inside __str__
> or __repr__.

I guess you are right, but still it is a pity. __str__ has been working for 
me beautifully for a long time now. The only change is that, previously, all 
the values had been read in or computed before calling __str__(), now I am 
delaying the computation until requested.

It is a bit like quantum theory. I have no way of telling whether the 
computation has been carried out without looking at it, but the act of 
looking at it triggers the computation. I can tell, of course, by looking at 
the underlying attribute, but not by using the public methods.

Frank




From rosuav at gmail.com  Wed Nov 23 08:01:07 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 24 Nov 2016 00:01:07 +1100
Subject: Is this pythonic?
In-Reply-To: <o141us$il0$1@blaine.gmane.org>
References: <o13meh$p2g$1@blaine.gmane.org> <o141us$il0$1@blaine.gmane.org>
Message-ID: <CAPTjJmqGEwHPVyrR+Ti9bV=S5MsLt3nquF4TvE=xpeEs188yzQ@mail.gmail.com>

On Wed, Nov 23, 2016 at 11:27 PM, Frank Millman <frank at chagford.com> wrote:
>
> @Chris
>>
>> This strongly suggests that str(x) is the wrong way to get the
>> information. You shouldn't be doing database requests inside __str__
>> or __repr__.
>
>
> I guess you are right, but still it is a pity. __str__ has been working for
> me beautifully for a long time now. The only change is that, previously, all
> the values had been read in or computed before calling __str__(), now I am
> delaying the computation until requested.
>
> It is a bit like quantum theory. I have no way of telling whether the
> computation has been carried out without looking at it, but the act of
> looking at it triggers the computation. I can tell, of course, by looking at
> the underlying attribute, but not by using the public methods.

That makes sense. What you could do is have __repr__ do something like this:

def __repr__(self):
    if self.has_data:
        return "<%s: %r>" % (self.col_name, self.data)
    return "<%s: <not fetched> >" % self.col_name

I'm not sure that that would be appropriate for __str__, though; maybe
it could return the string of data if it exists, otherwise it could
fall back on __repr__?

ChrisA


From frank at chagford.com  Wed Nov 23 08:55:13 2016
From: frank at chagford.com (Frank Millman)
Date: Wed, 23 Nov 2016 15:55:13 +0200
Subject: Is this pythonic?
In-Reply-To: <CAPTjJmqGEwHPVyrR+Ti9bV=S5MsLt3nquF4TvE=xpeEs188yzQ@mail.gmail.com>
References: <o13meh$p2g$1@blaine.gmane.org> <o141us$il0$1@blaine.gmane.org>
 <CAPTjJmqGEwHPVyrR+Ti9bV=S5MsLt3nquF4TvE=xpeEs188yzQ@mail.gmail.com>
Message-ID: <o14740$1qg$1@blaine.gmane.org>

"Chris Angelico"  wrote in message 
news:CAPTjJmqGEwHPVyrR+Ti9bV=S5MsLt3nquF4TvE=xpeEs188yzQ at mail.gmail.com...

> On Wed, Nov 23, 2016 at 11:27 PM, Frank Millman <frank at chagford.com> 
> wrote:
> >
> > @Chris
> >>
> >> This strongly suggests that str(x) is the wrong way to get the
> >> information. You shouldn't be doing database requests inside __str__
> >> or __repr__.
> >
> >
> > I guess you are right, but still it is a pity. __str__ has been working 
> > for
> > me beautifully for a long time now. The only change is that, previously, 
> > all
> > the values had been read in or computed before calling __str__(), now I 
> > am
> > delaying the computation until requested.
> >
> > It is a bit like quantum theory. I have no way of telling whether the
> > computation has been carried out without looking at it, but the act of
> > looking at it triggers the computation. I can tell, of course, by 
> > looking at
> > the underlying attribute, but not by using the public methods.
>
> That makes sense. What you could do is have __repr__ do something like 
> this:
>
> def __repr__(self):
>     if self.has_data:
>         return "<%s: %r>" % (self.col_name, self.data)
>     return "<%s: <not fetched> >" % self.col_name
>
> I'm not sure that that would be appropriate for __str__, though; maybe
> it could return the string of data if it exists, otherwise it could
> fall back on __repr__?
>

Thanks for the ideas. I will have to experiment a bit.

There is a certain irony in all this. When I started using asyncio, I just 
converted the networking functions into coroutines and waited for it to 
stabilise. Then I wanted to extend it, and found that coroutines can only be 
called by other coroutines, and I had some long chains of function calls, so 
I backed off. Then I eventually bit the bullet, converted everything in the 
chain to a coroutine, and let it settle down again. I have done this a few 
times, and each time I sensed an improvement in the way that my entire 
application was beginning to 'flow' in an async manner, which was good. 
However, I have managed to avoid turning getval() into a coroutine, until 
now. Now I am ready to embrace the change, but this time it is Python that 
is tripping me up.

For the time being I will use 'print(await obj.__str__())', as this is a 
good compromise. Of course I don't have to use __str__, I can call it 
anything, so I will probably create a helper function to make it easy to 
call on any object.

One of the things that was deterring me from turning getval() into a 
coroutine was the inability to use a coroutine inside a comprehension. I see 
that Python 3.6 now allows this, so I must download a beta version and try 
it out.

Frank




From jones.dayton at gmail.com  Wed Nov 23 09:18:38 2016
From: jones.dayton at gmail.com (jones.dayton at gmail.com)
Date: Wed, 23 Nov 2016 06:18:38 -0800 (PST)
Subject: Quick help for a python newby, please
Message-ID: <1d38d19f-81e9-4f21-8805-aeea61e972cb@googlegroups.com>

I'm just learning, so please excuse my ignorance for what I know is a simple issue...

I'm writing a "Hello, World" type of script to see how things work in python3.  I'm asking for input to get a person's birthday, then I want to compare that to "today" and see how many days have past.

--code--
from datetime import datetime, date

    person = input("Enter your name: ")
    bmonth = int(input("Enter the numerical month you were born: "))
    bday = int(input("Enter the numerical day of the month you were born: "))
    byear = int(input("Enter the year you were born: "))

    today = date.today()
-- end --

I know to get what I want, I need to do something like:
d0 = date(2008, 8, 18)
d1 = date(2008, 9, 26)
delta = d0 - d1
print(delta.days)


but how do I replace the "2008, 8, 18" and "2008, 9, 26" with *my* values?  I've tried several things (which I can't remember all of) but usually end up with an error like this:

   ...
    delta = (b - a)
TypeError: unsupported operand type(s) for -: 'str' and 'str'


From tomuxiong at gmx.com  Wed Nov 23 09:42:25 2016
From: tomuxiong at gmx.com (Thomas Nyberg)
Date: Wed, 23 Nov 2016 09:42:25 -0500
Subject: Quick help for a python newby, please
In-Reply-To: <1d38d19f-81e9-4f21-8805-aeea61e972cb@googlegroups.com>
References: <1d38d19f-81e9-4f21-8805-aeea61e972cb@googlegroups.com>
Message-ID: <c4846b97-6ead-3f93-2071-106a76dcf6e2@gmx.com>

On 11/23/2016 09:18 AM, jones.dayton at gmail.com wrote:
>
> but how do I replace the "2008, 8, 18" and "2008, 9, 26" with *my* values?  I've tried several things (which I can't remember all of) but usually end up with an error like this:
>
Does this not work for you?

	d = date(byear, bmonth, bday)

Then you could write something like:

	diff = date.today() - d

Of course you'll need to a do a bit more work to deal with years (e.g. 
has your birthday happened this year yet or not?), but does this not fix 
your problem?

Cheers,
Thomas


From jones.dayton at gmail.com  Wed Nov 23 10:13:32 2016
From: jones.dayton at gmail.com (Dayton Jones)
Date: Wed, 23 Nov 2016 07:13:32 -0800 (PST)
Subject: Quick help for a python newby, please
In-Reply-To: <mailman.61.1479913569.3146.python-list@python.org>
References: <1d38d19f-81e9-4f21-8805-aeea61e972cb@googlegroups.com>
 <c4846b97-6ead-3f93-2071-106a76dcf6e2@gmx.com>
 <mailman.61.1479913569.3146.python-list@python.org>
Message-ID: <132714e8-696a-420f-a634-78697874088e@googlegroups.com>

ah...yes, but then how could I strip the rest (h:m:s) from it?

Enter the numerical month you were born: 3
Enter the numerical day of the month you were born: 30
Enter the year you were born: 1989
10100 days, 0:00:00


From as at sci.fi  Wed Nov 23 10:18:23 2016
From: as at sci.fi (Anssi Saari)
Date: Wed, 23 Nov 2016 17:18:23 +0200
Subject: Quick help for a python newby, please
References: <1d38d19f-81e9-4f21-8805-aeea61e972cb@googlegroups.com>
Message-ID: <vg3fumib4xs.fsf@coffee.modeemi.fi>

jones.dayton at gmail.com writes:

> but how do I replace the "2008, 8, 18" and "2008, 9, 26" with *my* values?  I've tried several things (which I can't remember all of) but usually end up with an error like this:

Something like this?

today = date.today()
birthday = date(byear, bmonth, bday)
delta = today - birthday
print(delta.days)




From walters.justin01 at gmail.com  Wed Nov 23 10:57:15 2016
From: walters.justin01 at gmail.com (justin walters)
Date: Wed, 23 Nov 2016 07:57:15 -0800
Subject: Quick help for a python newby, please
In-Reply-To: <132714e8-696a-420f-a634-78697874088e@googlegroups.com>
References: <1d38d19f-81e9-4f21-8805-aeea61e972cb@googlegroups.com>
 <c4846b97-6ead-3f93-2071-106a76dcf6e2@gmx.com>
 <mailman.61.1479913569.3146.python-list@python.org>
 <132714e8-696a-420f-a634-78697874088e@googlegroups.com>
Message-ID: <CAO1D73G=AqBwnBvtT9Rxz6=yeTUv+7OaH=tEMfK=R6RvyNs2Pg@mail.gmail.com>

On Wed, Nov 23, 2016 at 7:13 AM, Dayton Jones <jones.dayton at gmail.com>
wrote:

> ah...yes, but then how could I strip the rest (h:m:s) from it?
>
> Enter the numerical month you were born: 3
> Enter the numerical day of the month you were born: 30
> Enter the year you were born: 1989
> 10100 days, 0:00:00
> --
> https://mail.python.org/mailman/listinfo/python-list
>


You'll probably want to use `timedelta`:
https://docs.python.org/3.4/library/datetime.html#timedelta-objects


From jones.dayton at gmail.com  Wed Nov 23 11:27:53 2016
From: jones.dayton at gmail.com (Dayton Jones)
Date: Wed, 23 Nov 2016 08:27:53 -0800 (PST)
Subject: Quick help for a python newby, please
In-Reply-To: <mailman.62.1479918010.3146.python-list@python.org>
References: <1d38d19f-81e9-4f21-8805-aeea61e972cb@googlegroups.com>
 <vg3fumib4xs.fsf@coffee.modeemi.fi>
 <mailman.62.1479918010.3146.python-list@python.org>
Message-ID: <d70d2d07-34ec-4d63-ad7e-927b542f8551@googlegroups.com>

Ah, perfect - thanks.  I was overly complicating things.


From daiyueweng at gmail.com  Wed Nov 23 12:40:44 2016
From: daiyueweng at gmail.com (Daiyue Weng)
Date: Wed, 23 Nov 2016 17:40:44 +0000
Subject: How to concatenate strings in different lists
Message-ID: <CAF_EDktLXuGTxsV=HnshQAuX9vYgsOp4hgykPkgxE2qztcg7Ew@mail.gmail.com>

Hi, I am wondering how to concatenate corresponding strings in two lists,
assuming that the lists are of same length, e.g.

val_1 = ['a', 'b', 'c']
val_2 = ['A', 'B', 'C']

# concatenate corresponding strings in 'val_1' and 'val_2'
# and put results in another list 'val' so that
# val = ['aA', 'bB', 'cC']

cheers


From liik.joonas at gmail.com  Wed Nov 23 12:53:18 2016
From: liik.joonas at gmail.com (Joonas Liik)
Date: Wed, 23 Nov 2016 19:53:18 +0200
Subject: How to concatenate strings in different lists
In-Reply-To: <CAF_EDktLXuGTxsV=HnshQAuX9vYgsOp4hgykPkgxE2qztcg7Ew@mail.gmail.com>
References: <CAF_EDktLXuGTxsV=HnshQAuX9vYgsOp4hgykPkgxE2qztcg7Ew@mail.gmail.com>
Message-ID: <CAB1GNpRQoMPDBsrJzeTfypFo89LBfgQKu-TVjx=wnsxLoHiGAw@mail.gmail.com>

On 23 November 2016 at 19:40, Daiyue Weng <daiyueweng at gmail.com> wrote:
> Hi, I am wondering how to concatenate corresponding strings in two lists,
> assuming that the lists are of same length, e.g.
>
> val_1 = ['a', 'b', 'c']
> val_2 = ['A', 'B', 'C']
>
> # concatenate corresponding strings in 'val_1' and 'val_2'
> # and put results in another list 'val' so that
> # val = ['aA', 'bB', 'cC']
>
> cheers
> --
> https://mail.python.org/mailman/listinfo/python-list

one way to do it:

val = [''.join(pair) for pair in zip(val_1, val2)]


From alxdoutsinis at gmail.com  Wed Nov 23 12:56:44 2016
From: alxdoutsinis at gmail.com (Alex)
Date: Wed, 23 Nov 2016 09:56:44 -0800 (PST)
Subject: K-means Python code analyse
Message-ID: <37e52421-dfc1-4186-a0fc-0905443abacb@googlegroups.com>

Can please anyone explaine me what do each of these code lines and how k-means algorithm works?


from scipy.cluster.vq import *
from scipy.misc import imresize
from pylab import *
from PIL import Image
steps = 500 
im = array(Image.open("empire.jpg"))
dx = im.shape[0] / steps
dy = im.shape[1] / steps
# compute color features for each region
features = []
for x in range(steps): 
    for y in range(steps):
        R = mean(im[x*dx:(x+1)*dx,y*dy:(y+1)*dy,0])
        G = mean(im[x*dx:(x+1)*dx,y*dy:(y+1)*dy,1])
        B = mean(im[x*dx:(x+1)*dx,y*dy:(y+1)*dy,2])
        features.append([R,G,B]) 
features = array(features,"f") # make into array
# cluster
centroids,variance = kmeans(features,3)
code,distance = vq(features,centroids)
# create image with cluster labels
codeim = code.reshape(steps,steps)
codeim = imresize(codeim,im.shape[:2],interp="nearest")
figure()
imshow(codeim)
show()


From jones.dayton at gmail.com  Wed Nov 23 13:02:45 2016
From: jones.dayton at gmail.com (Dayton Jones)
Date: Wed, 23 Nov 2016 10:02:45 -0800 (PST)
Subject: Can I print 2 calendars side by side?
Message-ID: <e2e71314-5543-48aa-b426-4369b7314bdb@googlegroups.com>

I'd like to be able to display 2 calendars side by side, instead of stacked... is this possible?

for instance:

    print(calendar.month(year_a,month))
    print()
    print(calendar.month(year_b,month))

prints:
     June 1971
Mo Tu We Th Fr Sa Su
    1  2  3  4  5  6
 7  8  9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30


     June 2017
Mo Tu We Th Fr Sa Su
          1  2  3  4
 5  6  7  8  9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30



but what I would like is:
     June 1971               June 2017
Mo Tu We Th Fr Sa Su    Mo Tu We Th Fr Sa Su
    1  2  3  4  5  6              1  2  3  4
 7  8  9 10 11 12 13     5  6  7  8  9 10 11
14 15 16 17 18 19 20    12 13 14 15 16 17 18
21 22 23 24 25 26 27    19 20 21 22 23 24 25
28 29 30                26 27 28 29 30


From tomuxiong at gmx.com  Wed Nov 23 13:03:14 2016
From: tomuxiong at gmx.com (Thomas Nyberg)
Date: Wed, 23 Nov 2016 13:03:14 -0500
Subject: How to concatenate strings in different lists
In-Reply-To: <CAF_EDktLXuGTxsV=HnshQAuX9vYgsOp4hgykPkgxE2qztcg7Ew@mail.gmail.com>
References: <CAF_EDktLXuGTxsV=HnshQAuX9vYgsOp4hgykPkgxE2qztcg7Ew@mail.gmail.com>
Message-ID: <c56ae9cd-1f04-04e9-1620-e1999da778e7@gmx.com>

On 11/23/2016 12:40 PM, Daiyue Weng wrote:
> Hi, I am wondering how to concatenate corresponding strings in two lists,
> assuming that the lists are of same length, e.g.
>
> val_1 = ['a', 'b', 'c']
> val_2 = ['A', 'B', 'C']
>
> # concatenate corresponding strings in 'val_1' and 'val_2'
> # and put results in another list 'val' so that
> # val = ['aA', 'bB', 'cC']
>
> cheers
>
Hello,

The easiest is probably first to zip them (i.e. pair the corresponding 
pieces together) and then to join them as you normally would with lists 
of strings:

 >>> print([''.join(pair) for pair in zip(val_1, val_2)])
['aA', 'bB', 'cC']

That uses list comprehensions which might be a bit confusing. In two 
steps, you could do the following

 >>> pairs = zip(val_1, val_2)
 >>> catted_strings = []
 >>> for pair in pairs:
         catted_string = ''.join(pair)
         catted_strings.append(catted_string)
 >>> print(catted_strings)
['aA', 'bB', 'cC']

Hope this this helps.

Cheers,
Thomas


From python.list at tim.thechases.com  Wed Nov 23 13:39:19 2016
From: python.list at tim.thechases.com (Tim Chase)
Date: Wed, 23 Nov 2016 12:39:19 -0600
Subject: Can I print 2 calendars side by side?
In-Reply-To: <e2e71314-5543-48aa-b426-4369b7314bdb@googlegroups.com>
References: <e2e71314-5543-48aa-b426-4369b7314bdb@googlegroups.com>
Message-ID: <20161123123919.19321070@bigbox.christie.dr>

On 2016-11-23 10:02, Dayton Jones wrote:
> I'd like to be able to display 2 calendars side by side, instead of
> stacked... is this possible?
> 
> for instance:
> 
>     print(calendar.month(year_a,month))
>     print()
>     print(calendar.month(year_b,month))
> 
> prints:
>      June 1971
> Mo Tu We Th Fr Sa Su
>     1  2  3  4  5  6
>  7  8  9 10 11 12 13
> 14 15 16 17 18 19 20
> 21 22 23 24 25 26 27
> 28 29 30
> 
> 
>      June 2017
> Mo Tu We Th Fr Sa Su
>           1  2  3  4
>  5  6  7  8  9 10 11
> 12 13 14 15 16 17 18
> 19 20 21 22 23 24 25
> 26 27 28 29 30
> 
> 
> but what I would like is:
>      June 1971               June 2017
> Mo Tu We Th Fr Sa Su    Mo Tu We Th Fr Sa Su
>     1  2  3  4  5  6              1  2  3  4
>  7  8  9 10 11 12 13     5  6  7  8  9 10 11
> 14 15 16 17 18 19 20    12 13 14 15 16 17 18
> 21 22 23 24 25 26 27    19 20 21 22 23 24 25
> 28 29 30                26 27 28 29 30

This seems to do the trick for me

  from itertools import izip_longest
  print ('\n'.join(
    "%-24s %s" % (week1, week2)
    for week1, week2
    in izip_longest(
      month(1971, 6).splitlines(),
      month(2017, 6).splitlines(),
      fillvalue='',
      )
    ))

Adjust the "-24" to put your desired amount of padding between the
calendars (needs to be at least 21=3 character-columns per day * 7
days per week).  If you want more than two weeks, you can generalize
it:

  dates = [
    month(1971, 6),
    month(2001, 6),
    month(2017, 6),
    ]

  print ('\n'.join(
    '  '.join(w.ljust(21) for w in weeks)
    for weeks
    in izip_longest(
      *[cal.splitlines() for cal in dates],
      fillvalue=''
      )
    ))

Hope this helps,

-tkc





From __peter__ at web.de  Wed Nov 23 13:57:41 2016
From: __peter__ at web.de (Peter Otten)
Date: Wed, 23 Nov 2016 19:57:41 +0100
Subject: Can I print 2 calendars side by side?
References: <e2e71314-5543-48aa-b426-4369b7314bdb@googlegroups.com>
Message-ID: <o14or2$7e2$1@blaine.gmane.org>

Dayton Jones wrote:

> I'd like to be able to display 2 calendars side by side, instead of
> stacked... is this possible?

I'm too lazy to look around for a ready-to-use solution, so here's my own:

$ cat side_by_side.py
from itertools import zip_longest

def zip_lines(*columns, sep="  "):
    columns = [c.splitlines() for c in columns]
    widths = [max(len(s) for s in c) for c in columns]
    return "\n".join(
        sep.join(c.ljust(w) for c, w in zip(row, widths))
        for row in zip_longest(*columns, fillvalue="")
    )

if __name__ == "__main__":
    import calendar

    print(zip_lines(calendar.month(2016, 1), calendar.month(2015, 1)))
    print()

    columns = [calendar.month(2016, month) for month in range(6, 9)]
    print(zip_lines(*columns, sep="  |  "))
$ python3 side_by_side.py 
    January 2016          January 2015    
Mo Tu We Th Fr Sa Su  Mo Tu We Th Fr Sa Su
             1  2  3            1  2  3  4
 4  5  6  7  8  9 10   5  6  7  8  9 10 11
11 12 13 14 15 16 17  12 13 14 15 16 17 18
18 19 20 21 22 23 24  19 20 21 22 23 24 25
25 26 27 28 29 30 31  26 27 28 29 30 31   

     June 2016        |       July 2016        |      August 2016     
Mo Tu We Th Fr Sa Su  |  Mo Tu We Th Fr Sa Su  |  Mo Tu We Th Fr Sa Su
       1  2  3  4  5  |               1  2  3  |   1  2  3  4  5  6  7
 6  7  8  9 10 11 12  |   4  5  6  7  8  9 10  |   8  9 10 11 12 13 14
13 14 15 16 17 18 19  |  11 12 13 14 15 16 17  |  15 16 17 18 19 20 21
20 21 22 23 24 25 26  |  18 19 20 21 22 23 24  |  22 23 24 25 26 27 28
27 28 29 30           |  25 26 27 28 29 30 31  |  29 30 31            




From python at mrabarnett.plus.com  Wed Nov 23 14:01:43 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Wed, 23 Nov 2016 19:01:43 +0000
Subject: Can I print 2 calendars side by side?
In-Reply-To: <e2e71314-5543-48aa-b426-4369b7314bdb@googlegroups.com>
References: <e2e71314-5543-48aa-b426-4369b7314bdb@googlegroups.com>
Message-ID: <6ee801bb-1b0a-11c6-8a3d-c3b7061ae389@mrabarnett.plus.com>

On 2016-11-23 18:02, Dayton Jones wrote:
> I'd like to be able to display 2 calendars side by side, instead of stacked... is this possible?
>
> for instance:
>
>     print(calendar.month(year_a,month))
>     print()
>     print(calendar.month(year_b,month))
>
> prints:
>      June 1971
> Mo Tu We Th Fr Sa Su
>     1  2  3  4  5  6
>  7  8  9 10 11 12 13
> 14 15 16 17 18 19 20
> 21 22 23 24 25 26 27
> 28 29 30
>
>
>      June 2017
> Mo Tu We Th Fr Sa Su
>           1  2  3  4
>  5  6  7  8  9 10 11
> 12 13 14 15 16 17 18
> 19 20 21 22 23 24 25
> 26 27 28 29 30
>
>
>
> but what I would like is:
>      June 1971               June 2017
> Mo Tu We Th Fr Sa Su    Mo Tu We Th Fr Sa Su
>     1  2  3  4  5  6              1  2  3  4
>  7  8  9 10 11 12 13     5  6  7  8  9 10 11
> 14 15 16 17 18 19 20    12 13 14 15 16 17 18
> 21 22 23 24 25 26 27    19 20 21 22 23 24 25
> 28 29 30                26 27 28 29 30
>
The functions return strings, so you can split them into lines:

rows_a = calendar.month(year_a, month).splitlines()
rows_b = calendar.month(year_b, month).splitlines()

You can then zip them together:

rows = ['{}    {}'.format(row_a, row_b) for row_a, row_b in zip(rows_a, 
rows_b)]

However:

1. The 2 months might have a different number of rows. 'zip' will 
truncate to the shorter one, so use itertools.zip_longest instead (give 
it a fillvalue of '').

2. The lines aren't all the same length, so you'll need to pad the 
left-hand one to the correct length (20 characters) to make everything 
line-up.



From jones.dayton at gmail.com  Wed Nov 23 14:12:45 2016
From: jones.dayton at gmail.com (Dayton Jones)
Date: Wed, 23 Nov 2016 11:12:45 -0800 (PST)
Subject: Can I print 2 calendars side by side?
In-Reply-To: <mailman.69.1479927475.3146.python-list@python.org>
References: <e2e71314-5543-48aa-b426-4369b7314bdb@googlegroups.com>
 <o14or2$7e2$1@blaine.gmane.org>
 <mailman.69.1479927475.3146.python-list@python.org>
Message-ID: <3db63bd8-9945-4435-bd72-40b792f706aa@googlegroups.com>

Perfect!  Thank you.


From twgrops at googlemail.com  Wed Nov 23 14:17:40 2016
From: twgrops at googlemail.com (Thomas Grops)
Date: Wed, 23 Nov 2016 11:17:40 -0800 (PST)
Subject: Random number help
Message-ID: <63c998bf-ff13-4e95-9b42-7d7c50d9a84c@googlegroups.com>

I need a way of generating a random number but there is a catch:

I don't want to include certain numbers, is this possible?

random.randint(1,100) works as it will randomly pick numbers between 1 and 100 but say i don't want 48 to come out is there a way of doing this. It needs to be an integer too so not a list unless there is a way to convert list to int

Many Thanks Tom


From marko at pacujo.net  Wed Nov 23 14:21:14 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Wed, 23 Nov 2016 21:21:14 +0200
Subject: Random number help
References: <63c998bf-ff13-4e95-9b42-7d7c50d9a84c@googlegroups.com>
Message-ID: <87k2buatp1.fsf@elektro.pacujo.net>

Thomas Grops <twgrops at googlemail.com>:

> random.randint(1,100) works as it will randomly pick numbers between 1
> and 100 but say i don't want 48 to come out is there a way of doing
> this. It needs to be an integer too so not a list unless there is a way
> to convert list to int

   r = 48
   while r == 48:
       r = random.randint(1, 100)


Marko


From ckaynor at zindagigames.com  Wed Nov 23 14:29:43 2016
From: ckaynor at zindagigames.com (Chris Kaynor)
Date: Wed, 23 Nov 2016 11:29:43 -0800
Subject: Random number help
In-Reply-To: <63c998bf-ff13-4e95-9b42-7d7c50d9a84c@googlegroups.com>
References: <63c998bf-ff13-4e95-9b42-7d7c50d9a84c@googlegroups.com>
Message-ID: <CALvWhxtf0Hwr+1n6ujfxeGprXLkH+-Cr2-pJ78J+8hqrEHxNtw@mail.gmail.com>

On Wed, Nov 23, 2016 at 11:17 AM, Thomas Grops via Python-list
<python-list at python.org> wrote:
> I need a way of generating a random number but there is a catch:
>
> I don't want to include certain numbers, is this possible?
>
> random.randint(1,100) works as it will randomly pick numbers between 1 and 100 but say i don't want 48 to come out is there a way of doing this. It needs to be an integer too so not a list unless there is a way to convert list to int

There are a few ways to accomplish this, depending on the exact
requirements. Here are some basic ideas:
- Generate a list of all valid values, and take the one at a random
index. random.sample may be useful. This is guaranteed to complete
(and in only one try), but generally takes extra memory. It is also a
reasonably easy way to sample without replacement (remove the picked
item each time).
- Generate a random number from the larger range, and retry if you get
one in the invalid set. Basically akin to rolling a die, and rerolling
until you don't get a 6. This could theoretically take forever,
however it is often good enough in practice. This will become more and
more inefficient as set of invalid values grows (especially if the
invalid set is a superset of the whole range)


From tomuxiong at gmx.com  Wed Nov 23 14:33:54 2016
From: tomuxiong at gmx.com (Thomas Nyberg)
Date: Wed, 23 Nov 2016 14:33:54 -0500
Subject: Random number help
In-Reply-To: <63c998bf-ff13-4e95-9b42-7d7c50d9a84c@googlegroups.com>
References: <63c998bf-ff13-4e95-9b42-7d7c50d9a84c@googlegroups.com>
Message-ID: <fa1edc04-cace-042c-f720-9d8f19340c47@gmx.com>

On 11/23/2016 02:17 PM, Thomas Grops via Python-list wrote:
> I need a way of generating a random number but there is a catch:
>
> I don't want to include certain numbers, is this possible?
>
> random.randint(1,100) works as it will randomly pick numbers between 1 and 100 but say i don't want 48 to come out is there a way of doing this. It needs to be an integer too so not a list unless there is a way to convert list to int
>
> Many Thanks Tom
>
If you specifically want to exclude 48, you could create the list of 
acceptable numbers and make a random choice from it. For example:

 >>> import random
 >>> l = list(range(1, 48)) + list(range(49, 101))
 >>> random.choice(l)
64

Cheers,
Thomas


From achmichael at gmail.com  Wed Nov 23 14:34:26 2016
From: achmichael at gmail.com (Achilleas Michailidis)
Date: Wed, 23 Nov 2016 21:34:26 +0200
Subject: Random number help
In-Reply-To: <1479929284.2322.0@smtp.gmail.com>
References: <63c998bf-ff13-4e95-9b42-7d7c50d9a84c@googlegroups.com>
 <1479929284.2322.0@smtp.gmail.com>
Message-ID: <1479929666.2322.1@smtp.gmail.com>


> You can use a while statement until you get a number out of your 
> excluded numbers
> 
> excludedNumbers = [1,2,3,4]
> randomNumber = random.randomint(1,100)
> 
> while randomNumber in excludedNumbers:
> 	randomNumber = random.randomint(1,100)
> 
> After the while you have a number outside the excluded numbers





From twgrops at googlemail.com  Wed Nov 23 14:37:58 2016
From: twgrops at googlemail.com (Thomas Grops)
Date: Wed, 23 Nov 2016 11:37:58 -0800 (PST)
Subject: Random number help
In-Reply-To: <mailman.72.1479929407.3146.python-list@python.org>
References: <63c998bf-ff13-4e95-9b42-7d7c50d9a84c@googlegroups.com>
 <CALvWhxtf0Hwr+1n6ujfxeGprXLkH+-Cr2-pJ78J+8hqrEHxNtw@mail.gmail.com>
 <mailman.72.1479929407.3146.python-list@python.org>
Message-ID: <0614a317-3525-4708-8c1e-177bbf6c8f28@googlegroups.com>

On Wednesday, 23 November 2016 19:30:21 UTC, Chris Kaynor  wrote:
> On Wed, Nov 23, 2016 at 11:17 AM, Thomas Grops via Python-list
> <python-list at python.org> wrote:
> > I need a way of generating a random number but there is a catch:
> >
> > I don't want to include certain numbers, is this possible?
> >
> > random.randint(1,100) works as it will randomly pick numbers between 1 and 100 but say i don't want 48 to come out is there a way of doing this. It needs to be an integer too so not a list unless there is a way to convert list to int
> 
> There are a few ways to accomplish this, depending on the exact
> requirements. Here are some basic ideas:
> - Generate a list of all valid values, and take the one at a random
> index. random.sample may be useful. This is guaranteed to complete
> (and in only one try), but generally takes extra memory. It is also a
> reasonably easy way to sample without replacement (remove the picked
> item each time).
> - Generate a random number from the larger range, and retry if you get
> one in the invalid set. Basically akin to rolling a die, and rerolling
> until you don't get a 6. This could theoretically take forever,
> however it is often good enough in practice. This will become more and
> more inefficient as set of invalid values grows (especially if the
> invalid set is a superset of the whole range)

many thanks but am I missing something when I use a list it returns a value with the type list not integer? I need an integer


From twgrops at googlemail.com  Wed Nov 23 14:39:25 2016
From: twgrops at googlemail.com (Thomas Grops)
Date: Wed, 23 Nov 2016 11:39:25 -0800 (PST)
Subject: Random number help
In-Reply-To: <mailman.71.1479929389.3146.python-list@python.org>
References: <63c998bf-ff13-4e95-9b42-7d7c50d9a84c@googlegroups.com>
 <fa1edc04-cace-042c-f720-9d8f19340c47@gmx.com>
 <mailman.71.1479929389.3146.python-list@python.org>
Message-ID: <bc21451a-072c-4d7d-b269-1eb39f0caa89@googlegroups.com>

On Wednesday, 23 November 2016 19:30:04 UTC, Thomas Nyberg  wrote:
> On 11/23/2016 02:17 PM, Thomas Grops via Python-list wrote:
> > I need a way of generating a random number but there is a catch:
> >
> > I don't want to include certain numbers, is this possible?
> >
> > random.randint(1,100) works as it will randomly pick numbers between 1 and 100 but say i don't want 48 to come out is there a way of doing this. It needs to be an integer too so not a list unless there is a way to convert list to int
> >
> > Many Thanks Tom
> >
> If you specifically want to exclude 48, you could create the list of 
> acceptable numbers and make a random choice from it. For example:
> 
>  >>> import random
>  >>> l = list(range(1, 48)) + list(range(49, 101))
>  >>> random.choice(l)
> 64
> 
> Cheers,
> Thomas

Thankyou, I just tried this but it seems to be printing every value excluding 48, I am just after a single value


From python at mrabarnett.plus.com  Wed Nov 23 14:43:09 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Wed, 23 Nov 2016 19:43:09 +0000
Subject: Random number help
In-Reply-To: <CALvWhxtf0Hwr+1n6ujfxeGprXLkH+-Cr2-pJ78J+8hqrEHxNtw@mail.gmail.com>
References: <63c998bf-ff13-4e95-9b42-7d7c50d9a84c@googlegroups.com>
 <CALvWhxtf0Hwr+1n6ujfxeGprXLkH+-Cr2-pJ78J+8hqrEHxNtw@mail.gmail.com>
Message-ID: <a5bb0c61-4ab8-145a-2801-b3968d5e19a3@mrabarnett.plus.com>

On 2016-11-23 19:29, Chris Kaynor wrote:
> On Wed, Nov 23, 2016 at 11:17 AM, Thomas Grops via Python-list
> <python-list at python.org> wrote:
>> I need a way of generating a random number but there is a catch:
>>
>> I don't want to include certain numbers, is this possible?
>>
>> random.randint(1,100) works as it will randomly pick numbers between 1 and 100 but say i don't want 48 to come out is there a way of doing this. It needs to be an integer too so not a list unless there is a way to convert list to int
>
> There are a few ways to accomplish this, depending on the exact
> requirements. Here are some basic ideas:
> - Generate a list of all valid values, and take the one at a random
> index. random.sample may be useful. This is guaranteed to complete
> (and in only one try), but generally takes extra memory. It is also a
> reasonably easy way to sample without replacement (remove the picked
> item each time).
> - Generate a random number from the larger range, and retry if you get
> one in the invalid set. Basically akin to rolling a die, and rerolling
> until you don't get a 6. This could theoretically take forever,
> however it is often good enough in practice. This will become more and
> more inefficient as set of invalid values grows (especially if the
> invalid set is a superset of the whole range)
>
Another way is to map a shorter range to the desired numbers using an 
offset:

r = random.randint(1, 99)
if r >= 48:
     r += 1



From twgrops at googlemail.com  Wed Nov 23 14:54:57 2016
From: twgrops at googlemail.com (Thomas Grops)
Date: Wed, 23 Nov 2016 11:54:57 -0800 (PST)
Subject: Random number help
In-Reply-To: <63c998bf-ff13-4e95-9b42-7d7c50d9a84c@googlegroups.com>
References: <63c998bf-ff13-4e95-9b42-7d7c50d9a84c@googlegroups.com>
Message-ID: <ea0ae740-14ba-4018-9641-2cd870c0e8f1@googlegroups.com>

Thankyou for all your help I have managed to pick a way that works from your suggestions :D


From greg.ewing at canterbury.ac.nz  Wed Nov 23 16:38:46 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Thu, 24 Nov 2016 10:38:46 +1300
Subject: Is this pythonic?
In-Reply-To: <mailman.60.1479909328.3146.python-list@python.org>
References: <o13meh$p2g$1@blaine.gmane.org> <o141us$il0$1@blaine.gmane.org>
 <CAPTjJmqGEwHPVyrR+Ti9bV=S5MsLt3nquF4TvE=xpeEs188yzQ@mail.gmail.com>
 <o14740$1qg$1@blaine.gmane.org>
 <mailman.60.1479909328.3146.python-list@python.org>
Message-ID: <e9mgj8FbcdqU1@mid.individual.net>

Frank Millman wrote:
> For the time being I will use 'print(await obj.__str__())', as this is a 
> good compromise.

It seems more like a very *bad* compromise to me.

I can't see how this gains you anything over just doing
print(await obj.getvalue()), and you lose the ability to
do anything that calls __str__ implicitly.

-- 
Greg


From steve+python at pearwood.info  Wed Nov 23 17:53:10 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Thu, 24 Nov 2016 09:53:10 +1100
Subject: Is this pythonic?
References: <o13meh$p2g$1@blaine.gmane.org> <o141us$il0$1@blaine.gmane.org>
 <mailman.54.1479904042.3146.python-list@python.org>
Message-ID: <58361dd7$0$1615$c3e8da3$5496439d@news.astraweb.com>

On Wed, 23 Nov 2016 11:27 pm, Frank Millman wrote:

> It is a bit like quantum theory. I have no way of telling whether the
> computation has been carried out without looking at it, but the act of
> looking at it triggers the computation. I can tell, of course, by looking
> at the underlying attribute, but not by using the public methods.

Then give it a public method (or better, a property) to tell.


@property
def needs_computation(self):
    return not hasattr(self, '_cached_value')




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From best_lay at yahoo.com  Wed Nov 23 18:02:32 2016
From: best_lay at yahoo.com (Wildman)
Date: Wed, 23 Nov 2016 17:02:32 -0600
Subject: Quick help for a python newby, please
References: <1d38d19f-81e9-4f21-8805-aeea61e972cb@googlegroups.com>
Message-ID: <-6SdnWQTVt2VvavFnZ2dnUU7-KGdnZ2d@giganews.com>

On Wed, 23 Nov 2016 06:18:38 -0800, jones.dayton wrote:

> I'm just learning, so please excuse my ignorance for
> what I know is a simple issue...
> 
> I'm writing a "Hello, World" type of script to see how
> things work in python3.  I'm asking for input to get a
> person's birthday, then I want to compare that to
> "today" and see how many days have past.
> 
> --code--
> from datetime import datetime, date
> 
>     person = input("Enter your name: ")
>     bmonth = int(input("Enter the numerical month you were born: "))
>     bday = int(input("Enter the numerical day of the month you were born: "))
>     byear = int(input("Enter the year you were born: "))
> 
>     today = date.today()
> -- end --
> 
> I know to get what I want, I need to do something like:
> d0 = date(2008, 8, 18)
> d1 = date(2008, 9, 26)
> delta = d0 - d1
> print(delta.days)
> 
> 
> but how do I replace the "2008, 8, 18" and "2008, 9, 26" with *my* values?  I've tried several things (which I can't remember all of) but usually end up with an error like this:
> 
>    ...
>     delta = (b - a)
> TypeError: unsupported operand type(s) for -: 'str' and 'str'

Try the code that is below:

import datetime
from datetime import date

today = date.today()
person = input("Enter your name: ")
byear = raw_input("Enter the four-digit year you were born: ")
bmonth = raw_input("Enter the month (1-12)you were born: ")
bday = raw_input("Enter the day (1-31) you were born: ")
try:
    birthdate = date(int(byear), int(bmonth), int(bday))
    dif = (today - birthdate).days
    print person + "'s birthday was " + str(dif) + " days ago."
except ValueError:
    print "The date you entered is not valid!"

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From none at invalid.com  Wed Nov 23 18:31:38 2016
From: none at invalid.com (mm0fmf)
Date: Wed, 23 Nov 2016 23:31:38 +0000
Subject: Reposting On Python-List PROHIBITED
In-Reply-To: <c3a1daed-e49f-468f-b290-0f93b84a6fa2@googlegroups.com>
References: <c3a1daed-e49f-468f-b290-0f93b84a6fa2@googlegroups.com>
Message-ID: <o158qg$n9j$1@dont-email.me>

On 23/11/2016 23:28, Lawrence D?Oliveiro wrote:
> To the controllers of Python-List:
>
> I never asked to be on your list. I never consented to having my postings appear on your list. And I certainly didn?t agree to any conditions you might impose on members on your list.
>
> Therefore, to see you take offence at something I said, and use that as an excuse to ?ban? me from your list, is an act of breathtaking hypocrisy. As far as I am concerned, it is you lot who should be banned from comp.lang.python. You are reusing its content inappropriately and without authorization. What you are doing must be stopped.
>

*plonk*


From rosuav at gmail.com  Wed Nov 23 19:59:17 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 24 Nov 2016 11:59:17 +1100
Subject: Quick help for a python newby, please
In-Reply-To: <-6SdnWQTVt2VvavFnZ2dnUU7-KGdnZ2d@giganews.com>
References: <1d38d19f-81e9-4f21-8805-aeea61e972cb@googlegroups.com>
 <-6SdnWQTVt2VvavFnZ2dnUU7-KGdnZ2d@giganews.com>
Message-ID: <CAPTjJmrr5OpVy8iVoApqM7q24Ad6=V4-9REK2kSV5PzU-rvHyw@mail.gmail.com>

On Thu, Nov 24, 2016 at 10:02 AM, Wildman via Python-list
<python-list at python.org> wrote:
> Try the code that is below:
>
> import datetime
> from datetime import date
>
> today = date.today()
> person = input("Enter your name: ")
> byear = raw_input("Enter the four-digit year you were born: ")

Please take care of Python versions. The OP said Python 3, and you've
kept one input() call, but then you've used raw_input for the rest.
These should all be input() calls in Py3.

ChrisA


From torriem at gmail.com  Wed Nov 23 20:24:14 2016
From: torriem at gmail.com (Michael Torrie)
Date: Wed, 23 Nov 2016 18:24:14 -0700
Subject: Reposting On Python-List PROHIBITED
In-Reply-To: <o158qg$n9j$1@dont-email.me>
References: <c3a1daed-e49f-468f-b290-0f93b84a6fa2@googlegroups.com>
 <o158qg$n9j$1@dont-email.me>
Message-ID: <a44a79d4-ce40-85db-00f5-ab5d6ddb1736@gmail.com>

On 11/23/2016 04:31 PM, mm0fmf wrote:
> On 23/11/2016 23:28, Lawrence D?Oliveiro wrote:
>> Therefore, to see you take offence at something I said, and use
>> that as an excuse to ?ban? me from your list, is an act of
>> breathtaking hypocrisy. As far as I am concerned, it is you lot who
>> should be banned from comp.lang.python. You are reusing its content
>> inappropriately and without authorization. What you are doing must
>> be stopped.

Oh wow.

Well the poor fellow can rest easy knowing that stuff he posts to the
newsgroup is indeed not going to the mailing list.

Personally it wouldn't hurt my feelings if comp.lang.python was cut
loose from the mailing list entirely, though I know many posters would
disagree and others would desire to drop the mailing list part.


From steve+python at pearwood.info  Wed Nov 23 21:41:47 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Thu, 24 Nov 2016 13:41:47 +1100
Subject: Reposting On Python-List PROHIBITED
References: <c3a1daed-e49f-468f-b290-0f93b84a6fa2@googlegroups.com>
Message-ID: <5836536c$0$1603$c3e8da3$5496439d@news.astraweb.com>

On Thu, 24 Nov 2016 10:28 am, Lawrence D?Oliveiro wrote:

> To the controllers of Python-List:
> 
> I never asked to be on your list. I never consented to having my postings
> appear on your list. And I certainly didn?t agree to any conditions you
> might impose on members on your list.

comp.lang.python is set up as a mirror of the mailing list. By posting to
comp.lang.python, you are mailing the python-list at python.org mailing list.


> Therefore, to see you take offence at something I said, and use that as an
> excuse to ?ban? me from your list, is an act of breathtaking hypocrisy.

It really isn't. It is perhaps an act of ignorance on your part to fail to
realise that you are posting to the mailing list, although I believe you
have been told this before. Repeatedly.


> As 
> far as I am concerned, it is you lot who should be banned from
> comp.lang.python. You are reusing its content inappropriately and without
> authorization. What you are doing must be stopped.

This newsgroup is a mirror of the mailing list, not the other way around.



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Wed Nov 23 21:43:07 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Thu, 24 Nov 2016 13:43:07 +1100
Subject: Is this pythonic?
References: <o13meh$p2g$1@blaine.gmane.org>
 <mailman.48.1479892259.3146.python-list@python.org>
 <87inrer0dl.fsf@elektro.pacujo.net> <o13thn$b0m$1@blaine.gmane.org>
 <mailman.51.1479899527.3146.python-list@python.org>
Message-ID: <583653bb$0$1603$c3e8da3$5496439d@news.astraweb.com>

On Wed, 23 Nov 2016 10:11 pm, Frank Millman wrote:

> Gah! The law of unintended consequences strikes again!
> 
> As I mentioned, the class in question represents a database column. 

Yes, you mentioned that.

> A 
> separate class represents a database row. I have a __str__() method on the
> 'row' class that prints a nicely formatted representation of the object
> with all of its column objects and their values.

You didn't mention that, but it shouldn't matter.

 
> With the above changes, I had to turn getval() into a coroutine.

You what?

I'm gobsmacked by this assertion. Nobody else seems to have commented on
this, so perhaps I'm missing something, but this strikes me as astonishing.
Nothing in your earlier post even hinted that you were using coroutines or
async, and as sure as the day is long memoisation doesn't force you to
start.

Even if the computation of the memoised value is done asynchronously, you
can easily split the computation off to a separate method (as you already
talked about doing!) and make getval() block until it returns.


[...]
> I can say 'print(await obj.__str__())', and it works, but I lose the
> ability to include it in a larger print statement.

Any time you find yourself directly calling dunder methods, you're probably
doing it wrong. This is one of those times.




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From gengyangcai at gmail.com  Wed Nov 23 21:44:20 2016
From: gengyangcai at gmail.com (Cai Gengyang)
Date: Wed, 23 Nov 2016 18:44:20 -0800 (PST)
Subject: NameError
Message-ID: <223a00e3-1e8b-4ba3-9402-37a7f8ba9dcf@googlegroups.com>

>>> pygame.draw.rect(screen, BROWN, [60, 400, 30, 45])
Traceback (most recent call last):
File "<pyshell#3>", line 1, in <module>
pygame.draw.rect(screen, BROWN, [60, 400, 30, 45])
NameError: name 'pygame' is not defined

How to solve this error ?


From joel.goldstick at gmail.com  Wed Nov 23 21:53:57 2016
From: joel.goldstick at gmail.com (Joel Goldstick)
Date: Wed, 23 Nov 2016 21:53:57 -0500
Subject: NameError
In-Reply-To: <223a00e3-1e8b-4ba3-9402-37a7f8ba9dcf@googlegroups.com>
References: <223a00e3-1e8b-4ba3-9402-37a7f8ba9dcf@googlegroups.com>
Message-ID: <CAPM-O+ySA-o3Afh12NU6Ce0REC_pnoH7_we4AOE-NYe9pSqOTw@mail.gmail.com>

On Wed, Nov 23, 2016 at 9:44 PM, Cai Gengyang <gengyangcai at gmail.com> wrote:
>>>> pygame.draw.rect(screen, BROWN, [60, 400, 30, 45])
> Traceback (most recent call last):
> File "<pyshell#3>", line 1, in <module>
> pygame.draw.rect(screen, BROWN, [60, 400, 30, 45])
> NameError: name 'pygame' is not defined
>
> How to solve this error ?
> --
> https://mail.python.org/mailman/listinfo/python-list

have you imported pygame?

-- 
Joel Goldstick
http://joelgoldstick.com/blog
http://cc-baseballstats.info/stats/birthdays


From gengyangcai at gmail.com  Wed Nov 23 22:02:47 2016
From: gengyangcai at gmail.com (Cai Gengyang)
Date: Wed, 23 Nov 2016 19:02:47 -0800 (PST)
Subject: NameError
In-Reply-To: <mailman.77.1479956046.3146.python-list@python.org>
References: <223a00e3-1e8b-4ba3-9402-37a7f8ba9dcf@googlegroups.com>
 <CAPM-O+ySA-o3Afh12NU6Ce0REC_pnoH7_we4AOE-NYe9pSqOTw@mail.gmail.com>
 <mailman.77.1479956046.3146.python-list@python.org>
Message-ID: <079079bb-6abd-4660-be25-dd4f33165e35@googlegroups.com>

I tried to import pygame by using these commands ------------------------------------------------https://www.google.com.sg/#q=how+to+import+pygame

but this is the error I got :

CaiGengYangs-MacBook-Pro:~ CaiGengYang$ sudo apt-get install python-pygame
sudo: apt-get: command not found



On Thursday, November 24, 2016 at 10:54:20 AM UTC+8, Joel Goldstick wrote:
> On Wed, Nov 23, 2016 at 9:44 PM, Cai Gengyang <gengyangcai at gmail.com> wrote:
> >>>> pygame.draw.rect(screen, BROWN, [60, 400, 30, 45])
> > Traceback (most recent call last):
> > File "<pyshell#3>", line 1, in <module>
> > pygame.draw.rect(screen, BROWN, [60, 400, 30, 45])
> > NameError: name 'pygame' is not defined
> >
> > How to solve this error ?
> > --
> > https://mail.python.org/mailman/listinfo/python-list
> 
> have you imported pygame?
> 
> -- 
> Joel Goldstick
> http://joelgoldstick.com/blog
> http://cc-baseballstats.info/stats/birthdays


From best_lay at yahoo.com  Wed Nov 23 22:41:19 2016
From: best_lay at yahoo.com (Wildman)
Date: Wed, 23 Nov 2016 21:41:19 -0600
Subject: Quick help for a python newby, please
References: <1d38d19f-81e9-4f21-8805-aeea61e972cb@googlegroups.com>
 <-6SdnWQTVt2VvavFnZ2dnUU7-KGdnZ2d@giganews.com>
 <CAPTjJmrr5OpVy8iVoApqM7q24Ad6=V4-9REK2kSV5PzU-rvHyw@mail.gmail.com>
 <mailman.75.1479949165.3146.python-list@python.org>
Message-ID: <HKCdnQvoIYbC_KvFnZ2dnUU7-bOdnZ2d@giganews.com>

On Thu, 24 Nov 2016 11:59:17 +1100, Chris Angelico wrote:

> On Thu, Nov 24, 2016 at 10:02 AM, Wildman via Python-list
> <python-list at python.org> wrote:
>> Try the code that is below:
>>
>> import datetime
>> from datetime import date
>>
>> today = date.today()
>> person = input("Enter your name: ")
>> byear = raw_input("Enter the four-digit year you were born: ")
> 
> Please take care of Python versions. The OP said Python 3, and you've
> kept one input() call, but then you've used raw_input for the rest.
> These should all be input() calls in Py3.
> 
> ChrisA

Point taken.  I did miss the python3 part.

I switched to raw_input because it handles an empty
input.  An empty input would trigger the ValueError.
No doubt with the correct code the same or similar
could be done with input().  My lack of experience
caused me to look for simpler solution and perhaps
the wrong one.

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From rosuav at gmail.com  Wed Nov 23 22:49:27 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 24 Nov 2016 14:49:27 +1100
Subject: Quick help for a python newby, please
In-Reply-To: <HKCdnQvoIYbC_KvFnZ2dnUU7-bOdnZ2d@giganews.com>
References: <1d38d19f-81e9-4f21-8805-aeea61e972cb@googlegroups.com>
 <-6SdnWQTVt2VvavFnZ2dnUU7-KGdnZ2d@giganews.com>
 <CAPTjJmrr5OpVy8iVoApqM7q24Ad6=V4-9REK2kSV5PzU-rvHyw@mail.gmail.com>
 <mailman.75.1479949165.3146.python-list@python.org>
 <HKCdnQvoIYbC_KvFnZ2dnUU7-bOdnZ2d@giganews.com>
Message-ID: <CAPTjJmqwS5yt3Mmx10Y9NmHW2rwDt2_7AiA9LmiiW8pUv8aZ=g@mail.gmail.com>

On Thu, Nov 24, 2016 at 2:41 PM, Wildman via Python-list
<python-list at python.org> wrote:
> Point taken.  I did miss the python3 part.
>
> I switched to raw_input because it handles an empty
> input.  An empty input would trigger the ValueError.
> No doubt with the correct code the same or similar
> could be done with input().  My lack of experience
> caused me to look for simpler solution and perhaps
> the wrong one.

The exact same thing is true of input() in Python 3. In Python 2,
input() is the same as eval(raw_input()), so don't use it ever [1].
You can happily use input() in Py3, even if you get empty input.

I like to put this at the top of cross-version scripts:

try: input = raw_input
except NameError: pass

Then you can proceed to use input() without worries.

ChrisA

[1] Yes, I'm aware there are times when evalling the user's input is
what you want. In those cases, be explicit and use eval.


From spluque at gmail.com  Wed Nov 23 22:58:16 2016
From: spluque at gmail.com (Seb)
Date: Wed, 23 Nov 2016 21:58:16 -0600
Subject: generating list of files matching condition
Message-ID: <87fumh4jhj.fsf@otaria.sebmel.org>

Hello,

Given a list of files:

In [81]: ec_files[0:10]
Out[81]: 

[u'EC_20160604002000.csv',
 u'EC_20160604010000.csv',
 u'EC_20160604012000.csv',
 u'EC_20160604014000.csv',
 u'EC_20160604020000.csv']

where the numbers are are a timestamp with format %Y%m%d%H%M%S, I'd like
to generate a list of matching files for each 2-hr period in a 2-h
frequency time series.  Ultimately I'm using Pandas to read and handle
the data in each group of files.  For the task of generating the files
for each 2-hr period, I've done the following:

beg_tstamp = pd.to_datetime(ec_files[0][-18:-4],
                            format="%Y%m%d%H%M%S")
end_tstamp = pd.to_datetime(ec_files[-1][-18:-4],
                            format="%Y%m%d%H%M%S")
tstamp_win = pd.date_range(beg_tstamp, end_tstamp, freq="2H")

So tstamp_win is the 2-hr frequency time series spanning the timestamps
in the files in ec_files.

I've generated the list of matching files for each tstamp_win using a
comprehension:

win_files = []
for i, w in enumerate(tstamp_win):
    nextw = w + pd.Timedelta(2, "h")
    ifiles = [x for x in ec_files if
              pd.to_datetime(x[-18:-4], format="%Y%m%d%H%M%S") >= w and
              pd.to_datetime(x[-18:-4], format="%Y%m%d%H%M%S") < nextw]
    win_files.append(ifiles)

However, this is proving very slow, and was wondering whether there's a
better/faster way to do this.  Any tips would be appreciated.


-- 
Seb



From tomuxiong at gmx.com  Wed Nov 23 23:04:06 2016
From: tomuxiong at gmx.com (Thomas Nyberg)
Date: Wed, 23 Nov 2016 23:04:06 -0500
Subject: NameError
In-Reply-To: <079079bb-6abd-4660-be25-dd4f33165e35@googlegroups.com>
References: <223a00e3-1e8b-4ba3-9402-37a7f8ba9dcf@googlegroups.com>
 <CAPM-O+ySA-o3Afh12NU6Ce0REC_pnoH7_we4AOE-NYe9pSqOTw@mail.gmail.com>
 <mailman.77.1479956046.3146.python-list@python.org>
 <079079bb-6abd-4660-be25-dd4f33165e35@googlegroups.com>
Message-ID: <33bcdbba-9c5d-d884-366b-5897812afbd0@gmx.com>

On 11/23/2016 10:02 PM, Cai Gengyang wrote:
> I tried to import pygame by using these commands ------------------------------------------------https://www.google.com.sg/#q=how+to+import+pygame
>
> but this is the error I got :
>
> CaiGengYangs-MacBook-Pro:~ CaiGengYang$ sudo apt-get install python-pygame
> sudo: apt-get: command not found
>
Judging by your prompt it looks like you're on a Mac. The apt-get 
program is installed on some Linux distributions (probably your 
instructions are for Ubuntu). Here are some istructions for installing 
pygame for a Macbook:

	http://pygame.org/wiki/macintosh
	http://florian-berger.de/en/articles/installing-pygame-for-python-3-on-os-x/
	https://jamesfriend.com.au/installing-pygame-python-3-mac-os-yosemite

Here's some info about getting a package manager like apt for Mac:

	http://unix.stackexchange.com/questions/80711/how-to-install-apt-get-or-yum-on-mac-os-x

Hopefully this helps.

Cheers,
Thomas


From orgnut at yahoo.com  Wed Nov 23 23:23:49 2016
From: orgnut at yahoo.com (Larry Hudson)
Date: Wed, 23 Nov 2016 20:23:49 -0800
Subject: How to you convert list of tuples to string
In-Reply-To: <fc2236f6-783d-490f-afcf-830841ff0633@googlegroups.com>
References: <CACT3xuX0BM3txkDttjp8Nebn6DaDLUMyVQxuC0t3Mj27VHjc6w@mail.gmail.com>
 <CDC17C8A-463D-4247-B1BB-D1D46A6243E3@xs4all.nl>
 <mailman.37.1479833528.3146.python-list@python.org>
 <Tu6dnbB_-s4Vy6jFnZ2dnUU7-eHNnZ2d@giganews.com>
 <fc2236f6-783d-490f-afcf-830841ff0633@googlegroups.com>
Message-ID: <PJWdnT1RqZ_I9qvFnZ2dnUU7-XPNnZ2d@giganews.com>

On 11/23/2016 03:09 AM, Ned Batchelder wrote:
[snip...]
>> Or using the new string formatting syntax:
>>
>> msg = '{},{},{}:{}'.format(*item)
>>
>> The *item in the format() unpacks the tuple.
>
>
> "new" == "Introduced in 2.6, available since 2008" :)
>
> --Ned.
>

Of course.  I probably should have said "newer" or "other".    :-)

-- 
      -=- Larry -=-


From gengyangcai at gmail.com  Wed Nov 23 23:48:55 2016
From: gengyangcai at gmail.com (Cai Gengyang)
Date: Wed, 23 Nov 2016 20:48:55 -0800 (PST)
Subject: NameError
In-Reply-To: <mailman.80.1479960005.3146.python-list@python.org>
References: <223a00e3-1e8b-4ba3-9402-37a7f8ba9dcf@googlegroups.com>
 <CAPM-O+ySA-o3Afh12NU6Ce0REC_pnoH7_we4AOE-NYe9pSqOTw@mail.gmail.com>
 <mailman.77.1479956046.3146.python-list@python.org>
 <079079bb-6abd-4660-be25-dd4f33165e35@googlegroups.com>
 <33bcdbba-9c5d-d884-366b-5897812afbd0@gmx.com>
 <mailman.80.1479960005.3146.python-list@python.org>
Message-ID: <e4c9f90e-3ef0-41b1-b729-a48792ebada9@googlegroups.com>

Yea, using Mac 

Following the instructions here for Mac ---https://bitbucket.org/pygame/pygame/issues/82/homebrew-on-leopard-fails-to-install#comment-627494

GengYang Cai CaiGengYangs-MacBook-Pro:~ CaiGengYang$ brew install python
==> Installing dependencies for python: xz, pkg-config, readline, sqlite,
==> Installing python dependency: xz
==> Downloading https://homebrew.bintray.com/.../xz-5.2.2.yosemite.bottle.ta
######################################################################## 100.0%
==> Pouring xz-5.2.2.yosemite.bottle.tar.gz
? /usr/local/Cellar/xz/5.2.2: 91 files, 1.4M
==> Installing python dependency: pkg-config
==> Downloading https://homebrew.bintray.com/.../pkg-config-0.29.1_1.yosemit
######################################################################## 100.0%
==> Pouring pkg-config-0.29.1_1.yosemite.bottle.tar.gz
? /usr/local/Cellar/pkg-config/0.29.1_1: 10 files, 627.3K
==> Installing python dependency: readline
==> Downloading https://homebrew.bintray.com/.../readline-6.3.8.yosemite.bot
######################################################################## 100.0%
==> Pouring readline-6.3.8.yosemite.bottle.tar.gz
==> Caveats
This formula is keg-only, which means it was not symlinked into /usr/local.

OS X provides the BSD libedit library, which shadows libreadline.
In order to prevent conflicts when programs look for libreadline we are
defaulting this GNU Readline installation to keg-only.

Generally there are no consequences of this for you. If you build your
own software and it requires this formula, you'll need to add to your
build variables:

LDFLAGS: -L/usr/local/opt/readline/lib
CPPFLAGS: -I/usr/local/opt/readline/include

==> Summary
? /usr/local/Cellar/readline/6.3.8: 46 files, 2M
==> Installing python dependency: sqlite
==> Downloading https://homebrew.bintray.com/.../sqlite-3.13.0.yosemite.bott
######################################################################## 100.0%
==> Pouring sqlite-3.13.0.yosemite.bottle.tar.gz
==> Caveats
This formula is keg-only, which means it was not symlinked into /usr/local.

OS X provides an older sqlite3.

Generally there are no consequences of this for you. If you build your
own software and it requires this formula, you'll need to add to your
build variables:

LDFLAGS: -L/usr/local/opt/sqlite/lib
CPPFLAGS: -I/usr/local/opt/sqlite/include

==> Summary
? /usr/local/Cellar/sqlite/3.13.0: 10 files, 2.9M
==> Installing python dependency: gdbm
==> Downloading https://homebrew.bintray.com/.../gdbm-1.12.yosemite.bottle.t
######################################################################## 100.0%
==> Pouring gdbm-1.12.yosemite.bottle.tar.gz
? /usr/local/Cellar/gdbm/1.12: 18 files, 490.8K
==> Installing python dependency: openssl
==> Downloading https://homebrew.bintray.com/.../openssl-1.0.2h_1.yosemite.b
######################################################################## 100.0%
==> Pouring openssl-1.0.2h_1.yosemite.bottle.tar.gz
==> Caveats
A CA file has been bootstrapped using certificates from the system
keychain. To add additional certificates, place .pem files in
/usr/local/etc/openssl/certs

and run
/usr/local/opt/openssl/bin/c_rehash

This formula is keg-only, which means it was not symlinked into /usr/local.

Apple has deprecated use of OpenSSL in favor of its own TLS and crypto libraries

Generally there are no consequences of this for you. If you build your
own software and it requires this formula, you'll need to add to your
build variables:

LDFLAGS: -L/usr/local/opt/openssl/lib
CPPFLAGS: -I/usr/local/opt/openssl/include

==> Summary
? /usr/local/Cellar/openssl/1.0.2h_1: 1,691 files, 12.0M
==> Installing python
Warning: Building python from source:
The bottle needs the Apple Command Line Tools to be installed.
You can install them, if desired, with:
xcode-select --install

==> Downloading https://www.python.org/.../2.7.12/Python-2.7.12.tar.xz
######################################################################## 100.0%
==> Downloading https://bugs.python.org/file30805/issue10910-workaround.txt
######################################################################## 100.0%
==> Patching
==> Applying issue10910-workaround.txt
patching file Include/pyport.h
Hunk #1 succeeded at 713 (offset 14 lines).
Hunk #2 succeeded at 736 (offset 14 lines).
==> ./configure --prefix=/usr/local/Cellar/python/2.7.12 --enable-ipv6 --dataroo
==> make
/usr/local/share/python/easy_install mecurial

brew install sdl
brew install sdl_mixer
brew install sdl_ttf
brew install sdl_image

hg clone https://bitbucket.org/pygame/pygame
cd pygame
/usr/local/bin/python setup.py install


Does this work ?







On Thursday, November 24, 2016 at 12:00:18 PM UTC+8, Thomas Nyberg wrote:
> On 11/23/2016 10:02 PM, Cai Gengyang wrote:
> > I tried to import pygame by using these commands ------------------------------------------------https://www.google.com.sg/#q=how+to+import+pygame
> >
> > but this is the error I got :
> >
> > CaiGengYangs-MacBook-Pro:~ CaiGengYang$ sudo apt-get install python-pygame
> > sudo: apt-get: command not found
> >
> Judging by your prompt it looks like you're on a Mac. The apt-get 
> program is installed on some Linux distributions (probably your 
> instructions are for Ubuntu). Here are some istructions for installing 
> pygame for a Macbook:
> 
> 	http://pygame.org/wiki/macintosh
> 	http://florian-berger.de/en/articles/installing-pygame-for-python-3-on-os-x/
> 	https://jamesfriend.com.au/installing-pygame-python-3-mac-os-yosemite
> 
> Here's some info about getting a package manager like apt for Mac:
> 
> 	http://unix.stackexchange.com/questions/80711/how-to-install-apt-get-or-yum-on-mac-os-x
> 
> Hopefully this helps.
> 
> Cheers,
> Thomas


From frank at chagford.com  Wed Nov 23 23:55:52 2016
From: frank at chagford.com (Frank Millman)
Date: Thu, 24 Nov 2016 06:55:52 +0200
Subject: Is this pythonic?
In-Reply-To: <583653bb$0$1603$c3e8da3$5496439d@news.astraweb.com>
References: <o13meh$p2g$1@blaine.gmane.org>
 <mailman.48.1479892259.3146.python-list@python.org>
 <87inrer0dl.fsf@elektro.pacujo.net> <o13thn$b0m$1@blaine.gmane.org>
 <mailman.51.1479899527.3146.python-list@python.org>
 <583653bb$0$1603$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o15rsm$vpp$1@blaine.gmane.org>

"Steve D'Aprano"  wrote in message 
news:583653bb$0$1603$c3e8da3$5496439d at news.astraweb.com...

> Even if the computation of the memoised value is done asynchronously, you
> can easily split the computation off to a separate method (as you already
> talked about doing!) and make getval() block until it returns.

Surely that defeats the whole purpose of asyncio. Anything that blocks holds 
up the entire process. I strenuously try to avoid blocking in any shape or 
form.

> > I can say 'print(await obj.__str__())', and it works, but I lose the
> > ability to include it in a larger print statement.

> Any time you find yourself directly calling dunder methods, you're 
> probably
> doing it wrong. This is one of those times.

Yes. Having slept on it, I realise I over-reacted.

The __str__() method is convenient for me, but I only use it for testing and 
debugging to see what is going on. It is not part of my app per se.

I now realise that the solution is -

1. Keep the __str__ method, but replace calls to getval() with a direct 
reference to the underlying attribute. It means that any 'computable' 
objects that have not already been computed will return None, but that is ok 
for my purposes.

2. Write a separate method, retaining the calls to getval(), to be called 
independently using 'await' if I ever need to see the full result after 
computation.

Frank




From nathan.ernst at gmail.com  Thu Nov 24 00:22:32 2016
From: nathan.ernst at gmail.com (Nathan Ernst)
Date: Wed, 23 Nov 2016 23:22:32 -0600
Subject: NameError
In-Reply-To: <e4c9f90e-3ef0-41b1-b729-a48792ebada9@googlegroups.com>
References: <223a00e3-1e8b-4ba3-9402-37a7f8ba9dcf@googlegroups.com>
 <CAPM-O+ySA-o3Afh12NU6Ce0REC_pnoH7_we4AOE-NYe9pSqOTw@mail.gmail.com>
 <mailman.77.1479956046.3146.python-list@python.org>
 <079079bb-6abd-4660-be25-dd4f33165e35@googlegroups.com>
 <33bcdbba-9c5d-d884-366b-5897812afbd0@gmx.com>
 <mailman.80.1479960005.3146.python-list@python.org>
 <e4c9f90e-3ef0-41b1-b729-a48792ebada9@googlegroups.com>
Message-ID: <CAPjBxg9LrwO3t2VVEs9p+fTL8a6HuCLbyF3rbMAqOU+e=0wNGQ@mail.gmail.com>

I don't see anything in that output resembling an error, just a few
warnings that some features may no be available.

Have you tried importing pygame after you did that? That's what'll prove
one way or another that it worked.

Regards,
Nate

On Wed, Nov 23, 2016 at 10:48 PM, Cai Gengyang <gengyangcai at gmail.com>
wrote:

> Yea, using Mac
>
> Following the instructions here for Mac ---https://bitbucket.org/
> pygame/pygame/issues/82/homebrew-on-leopard-fails-to-
> install#comment-627494
>
> GengYang Cai CaiGengYangs-MacBook-Pro:~ CaiGengYang$ brew install python
> ==> Installing dependencies for python: xz, pkg-config, readline, sqlite,
> ==> Installing python dependency: xz
> ==> Downloading https://homebrew.bintray.com/.
> ../xz-5.2.2.yosemite.bottle.ta
> ########################################################################
> 100.0%
> ==> Pouring xz-5.2.2.yosemite.bottle.tar.gz
> ? /usr/local/Cellar/xz/5.2.2: 91 files, 1.4M
> ==> Installing python dependency: pkg-config
> ==> Downloading https://homebrew.bintray.com/.
> ../pkg-config-0.29.1_1.yosemit
> ########################################################################
> 100.0%
> ==> Pouring pkg-config-0.29.1_1.yosemite.bottle.tar.gz
> ? /usr/local/Cellar/pkg-config/0.29.1_1: 10 files, 627.3K
> ==> Installing python dependency: readline
> ==> Downloading https://homebrew.bintray.com/.
> ../readline-6.3.8.yosemite.bot
> ########################################################################
> 100.0%
> ==> Pouring readline-6.3.8.yosemite.bottle.tar.gz
> ==> Caveats
> This formula is keg-only, which means it was not symlinked into /usr/local.
>
> OS X provides the BSD libedit library, which shadows libreadline.
> In order to prevent conflicts when programs look for libreadline we are
> defaulting this GNU Readline installation to keg-only.
>
> Generally there are no consequences of this for you. If you build your
> own software and it requires this formula, you'll need to add to your
> build variables:
>
> LDFLAGS: -L/usr/local/opt/readline/lib
> CPPFLAGS: -I/usr/local/opt/readline/include
>
> ==> Summary
> ? /usr/local/Cellar/readline/6.3.8: 46 files, 2M
> ==> Installing python dependency: sqlite
> ==> Downloading https://homebrew.bintray.com/.
> ../sqlite-3.13.0.yosemite.bott
> ########################################################################
> 100.0%
> ==> Pouring sqlite-3.13.0.yosemite.bottle.tar.gz
> ==> Caveats
> This formula is keg-only, which means it was not symlinked into /usr/local.
>
> OS X provides an older sqlite3.
>
> Generally there are no consequences of this for you. If you build your
> own software and it requires this formula, you'll need to add to your
> build variables:
>
> LDFLAGS: -L/usr/local/opt/sqlite/lib
> CPPFLAGS: -I/usr/local/opt/sqlite/include
>
> ==> Summary
> ? /usr/local/Cellar/sqlite/3.13.0: 10 files, 2.9M
> ==> Installing python dependency: gdbm
> ==> Downloading https://homebrew.bintray.com/.
> ../gdbm-1.12.yosemite.bottle.t
> ########################################################################
> 100.0%
> ==> Pouring gdbm-1.12.yosemite.bottle.tar.gz
> ? /usr/local/Cellar/gdbm/1.12: 18 files, 490.8K
> ==> Installing python dependency: openssl
> ==> Downloading https://homebrew.bintray.com/.
> ../openssl-1.0.2h_1.yosemite.b
> ########################################################################
> 100.0%
> ==> Pouring openssl-1.0.2h_1.yosemite.bottle.tar.gz
> ==> Caveats
> A CA file has been bootstrapped using certificates from the system
> keychain. To add additional certificates, place .pem files in
> /usr/local/etc/openssl/certs
>
> and run
> /usr/local/opt/openssl/bin/c_rehash
>
> This formula is keg-only, which means it was not symlinked into /usr/local.
>
> Apple has deprecated use of OpenSSL in favor of its own TLS and crypto
> libraries
>
> Generally there are no consequences of this for you. If you build your
> own software and it requires this formula, you'll need to add to your
> build variables:
>
> LDFLAGS: -L/usr/local/opt/openssl/lib
> CPPFLAGS: -I/usr/local/opt/openssl/include
>
> ==> Summary
> ? /usr/local/Cellar/openssl/1.0.2h_1: 1,691 files, 12.0M
> ==> Installing python
> Warning: Building python from source:
> The bottle needs the Apple Command Line Tools to be installed.
> You can install them, if desired, with:
> xcode-select --install
>
> ==> Downloading https://www.python.org/.../2.7.12/Python-2.7.12.tar.xz
> ########################################################################
> 100.0%
> ==> Downloading https://bugs.python.org/file30805/issue10910-
> workaround.txt
> ########################################################################
> 100.0%
> ==> Patching
> ==> Applying issue10910-workaround.txt
> patching file Include/pyport.h
> Hunk #1 succeeded at 713 (offset 14 lines).
> Hunk #2 succeeded at 736 (offset 14 lines).
> ==> ./configure --prefix=/usr/local/Cellar/python/2.7.12 --enable-ipv6
> --dataroo
> ==> make
> /usr/local/share/python/easy_install mecurial
>
> brew install sdl
> brew install sdl_mixer
> brew install sdl_ttf
> brew install sdl_image
>
> hg clone https://bitbucket.org/pygame/pygame
> cd pygame
> /usr/local/bin/python setup.py install
>
>
> Does this work ?
>
>
>
>
>
>
>
> On Thursday, November 24, 2016 at 12:00:18 PM UTC+8, Thomas Nyberg wrote:
> > On 11/23/2016 10:02 PM, Cai Gengyang wrote:
> > > I tried to import pygame by using these commands
> ------------------------------------------------https://www.
> google.com.sg/#q=how+to+import+pygame
> > >
> > > but this is the error I got :
> > >
> > > CaiGengYangs-MacBook-Pro:~ CaiGengYang$ sudo apt-get install
> python-pygame
> > > sudo: apt-get: command not found
> > >
> > Judging by your prompt it looks like you're on a Mac. The apt-get
> > program is installed on some Linux distributions (probably your
> > instructions are for Ubuntu). Here are some istructions for installing
> > pygame for a Macbook:
> >
> >       http://pygame.org/wiki/macintosh
> >       http://florian-berger.de/en/articles/installing-pygame-
> for-python-3-on-os-x/
> >       https://jamesfriend.com.au/installing-pygame-python-3-
> mac-os-yosemite
> >
> > Here's some info about getting a package manager like apt for Mac:
> >
> >       http://unix.stackexchange.com/questions/80711/how-to-
> install-apt-get-or-yum-on-mac-os-x
> >
> > Hopefully this helps.
> >
> > Cheers,
> > Thomas
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From best_lay at yahoo.com  Thu Nov 24 00:46:07 2016
From: best_lay at yahoo.com (Wildman)
Date: Wed, 23 Nov 2016 23:46:07 -0600
Subject: Quick help for a python newby, please
References: <1d38d19f-81e9-4f21-8805-aeea61e972cb@googlegroups.com>
 <-6SdnWQTVt2VvavFnZ2dnUU7-KGdnZ2d@giganews.com>
 <CAPTjJmrr5OpVy8iVoApqM7q24Ad6=V4-9REK2kSV5PzU-rvHyw@mail.gmail.com>
 <mailman.75.1479949165.3146.python-list@python.org>
 <HKCdnQvoIYbC_KvFnZ2dnUU7-bOdnZ2d@giganews.com>
 <CAPTjJmqwS5yt3Mmx10Y9NmHW2rwDt2_7AiA9LmiiW8pUv8aZ=g@mail.gmail.com>
 <mailman.78.1479959370.3146.python-list@python.org>
Message-ID: <XOGdnTErGo0C46vFnZ2dnUU7-YudnZ2d@giganews.com>

On Thu, 24 Nov 2016 14:49:27 +1100, Chris Angelico wrote:

> On Thu, Nov 24, 2016 at 2:41 PM, Wildman via Python-list
> <python-list at python.org> wrote:
>> Point taken.  I did miss the python3 part.
>>
>> I switched to raw_input because it handles an empty
>> input.  An empty input would trigger the ValueError.
>> No doubt with the correct code the same or similar
>> could be done with input().  My lack of experience
>> caused me to look for simpler solution and perhaps
>> the wrong one.
> 
> The exact same thing is true of input() in Python 3. In Python 2,
> input() is the same as eval(raw_input()), so don't use it ever [1].
> You can happily use input() in Py3, even if you get empty input.
> 
> I like to put this at the top of cross-version scripts:
> 
> try: input = raw_input
> except NameError: pass
> 
> Then you can proceed to use input() without worries.
> 
> ChrisA
> 
> [1] Yes, I'm aware there are times when evalling the user's input is
> what you want. In those cases, be explicit and use eval.

Understood.  Thank you.

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From orgnut at yahoo.com  Thu Nov 24 01:02:30 2016
From: orgnut at yahoo.com (Larry Hudson)
Date: Wed, 23 Nov 2016 22:02:30 -0800
Subject: Random number help
In-Reply-To: <63c998bf-ff13-4e95-9b42-7d7c50d9a84c@googlegroups.com>
References: <63c998bf-ff13-4e95-9b42-7d7c50d9a84c@googlegroups.com>
Message-ID: <U7ednaPbX87rH6vFnZ2dnUU7-aPNnZ2d@giganews.com>

On 11/23/2016 11:17 AM, Thomas Grops wrote:
> I need a way of generating a random number but there is a catch:
>
> I don't want to include certain numbers, is this possible?
>
> random.randint(1,100) works as it will randomly pick numbers between 1 and 100 but say i don't want 48 to come out is there a way of doing this. It needs to be an integer too so not a list unless there is a way to convert list to int
>
> Many Thanks Tom
>

Here's a possible generic approach:

#  Come up with a better name for this function
def randx(lo, hi, nw):  #  nw is a list of not-wanted ints
     while True:
         n = random.randint(lo, hi)
         if n not in nw:
             return n

Use it the same way as randint(), plus a list of the not-wanted values.
Short example:

-------
nw = [1, 3, 5, 7, 9]  #  Odd numbers out
for i in range(10):
     print(randx(1, 10, nw), end=' ')
print()
-------

Here's the results I got for 3 runs...

4 4 4 8 10 6 8 2 4 8
8 4 2 4 6 8 2 4 8 8
10 6 6 4 4 4 8 2 8 4

Of course, the not-wanted list can be a single int, or even empty.

-- 
      -=- Larry -=-


From steve+comp.lang.python at pearwood.info  Thu Nov 24 01:06:13 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Thu, 24 Nov 2016 17:06:13 +1100
Subject: Is this pythonic?
References: <o13meh$p2g$1@blaine.gmane.org>
 <mailman.48.1479892259.3146.python-list@python.org>
 <87inrer0dl.fsf@elektro.pacujo.net> <o13thn$b0m$1@blaine.gmane.org>
 <mailman.51.1479899527.3146.python-list@python.org>
 <583653bb$0$1603$c3e8da3$5496439d@news.astraweb.com>
 <o15rsm$vpp$1@blaine.gmane.org>
 <mailman.81.1479963365.3146.python-list@python.org>
Message-ID: <58368358$0$1513$c3e8da3$5496439d@news.astraweb.com>

On Thursday 24 November 2016 15:55, Frank Millman wrote:

> "Steve D'Aprano"  wrote in message
> news:583653bb$0$1603$c3e8da3$5496439d at news.astraweb.com...
> 
>> Even if the computation of the memoised value is done asynchronously, you
>> can easily split the computation off to a separate method (as you already
>> talked about doing!) and make getval() block until it returns.
> 
> Surely that defeats the whole purpose of asyncio. Anything that blocks holds
> up the entire process. I strenuously try to avoid blocking in any shape or
> form.

Perhaps I'm not understanding your task correctly, but surely you have to wait 
for the computation to occur at some point? Even if that's just you hitting 
Refresh waiting for the value of the column to eventually show up.

I'm a newbie to asyncio, but if I were doing this using threads, I'd have 
getval() set the self._cached_value to "pending..." (say), start the 
computation thread running, and then return. The computation thread will 
eventually write the true value to _cached_value, and in the meantime the 
getval() method (and hence __str__ will happily use the "pending..." value. The 
only tricky part is to make sure you only start the thread once.



-- 
Steven
299792.458 km/s ? not just a good idea, it?s the law!



From frank at chagford.com  Thu Nov 24 02:33:14 2016
From: frank at chagford.com (Frank Millman)
Date: Thu, 24 Nov 2016 09:33:14 +0200
Subject: Is this pythonic?
In-Reply-To: <58368358$0$1513$c3e8da3$5496439d@news.astraweb.com>
References: <o13meh$p2g$1@blaine.gmane.org>
 <mailman.48.1479892259.3146.python-list@python.org>
 <87inrer0dl.fsf@elektro.pacujo.net> <o13thn$b0m$1@blaine.gmane.org>
 <mailman.51.1479899527.3146.python-list@python.org>
 <583653bb$0$1603$c3e8da3$5496439d@news.astraweb.com>
 <o15rsm$vpp$1@blaine.gmane.org>
 <mailman.81.1479963365.3146.python-list@python.org>
 <58368358$0$1513$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o1653p$65s$1@blaine.gmane.org>

"Steven D'Aprano"  wrote in message 
news:58368358$0$1513$c3e8da3$5496439d at news.astraweb.com...

> On Thursday 24 November 2016 15:55, Frank Millman wrote:
>
> > "Steve D'Aprano"  wrote in message
> > news:583653bb$0$1603$c3e8da3$5496439d at news.astraweb.com...
> >
> >> Even if the computation of the memoised value is done asynchronously, 
> >> you
> >> can easily split the computation off to a separate method (as you 
> >> already
> >> talked about doing!) and make getval() block until it returns.
> >
> > Surely that defeats the whole purpose of asyncio. Anything that blocks 
> > holds
> > up the entire process. I strenuously try to avoid blocking in any shape 
> > or
> > form.
>
> Perhaps I'm not understanding your task correctly, but surely you have to 
> wait
> for the computation to occur at some point? Even if that's just you 
> hitting
> Refresh waiting for the value of the column to eventually show up.
>
> I'm a newbie to asyncio, but if I were doing this using threads, I'd have
> getval() set the self._cached_value to "pending..." (say), start the
> computation thread running, and then return. The computation thread will
> eventually write the true value to _cached_value, and in the meantime the
> getval() method (and hence __str__ will happily use the "pending..." 
> value. The
> only tricky part is to make sure you only start the thread once.
>

I am not really qualified to answer this - I *use* asyncio, but I don?t 
really understand what goes on under the covers.

With that caveat, here goes.

To me, the beauty of asyncio (or I suppose async in general) is that I don't 
have to worry about any of what you describe above.

I just have to 'await' whatever I am waiting for. There could be a long 
chain of function calls (which I suppose I should call coroutine calls) but 
at some point one of them is actually going to wait for some I/O, and yield 
control back to the event loop.

At that point, the entire chain is suspended, pending return of the value. 
Once received, control passes back down the chain to the originating 
coroutine, which can then carry on exactly where it left off.

Frank




From dieter at handshake.de  Thu Nov 24 02:38:39 2016
From: dieter at handshake.de (dieter)
Date: Thu, 24 Nov 2016 08:38:39 +0100
Subject: Question about working with html entities in python 2 to use them
 as filenames
References: <f662e584-c49f-1249-acce-2306a384e655@chello.at>
 <bb91d053-8505-f235-067e-c5e895dcdb98@chello.at>
 <mailman.39.1479848070.3146.python-list@python.org>
 <o12f5d$64h$1@dont-email.me>
 <5834f1c7$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <9774898c-1d86-6529-bb21-0f9b40e2e9c5@chello.at>
Message-ID: <8760nds4xs.fsf@handshake.de>

Steven Truppe <steven.truppe at chello.at> writes:

> type= <type 'str'> title =  Wizo - Anderster Full Album - YouTube
> type= <type 'str'> title =  Wizo - Bleib Tapfer / f?r&#39;n Arsch Full
> Album - YouTube
> Traceback (most recent call last):
>   File "./music-fetcher.py", line 39, in <module>
>     title = HTMLParser.HTMLParser().unescape(title)
>   File "/usr/lib/python2.7/HTMLParser.py", line 475, in unescape
>     return re.sub(r"&(#?[xX]?(?:[0-9a-fA-F]+|\w{1,8}));",
> replaceEntities, s)
>   File "/usr/lib/python2.7/re.py", line 155, in sub
>     return _compile(pattern, flags).sub(repl, string, count)
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position
> 23: ordinal not in range(128)

This looks like a bug with "HTMLParser" or a usage problem with its
"unescape" method.

I would use "lxml" in order to parse your HTML. It automatically converts
character references (like the above "&39;") and handles special
characters (like "?") adequately. Under Python 2, "lxml" either returns text
data as "str" (if the result is fully ascii) or "unicode" (otherwise).



From dieter at handshake.de  Thu Nov 24 02:41:44 2016
From: dieter at handshake.de (dieter)
Date: Thu, 24 Nov 2016 08:41:44 +0100
Subject: K-means Python code analyse
References: <37e52421-dfc1-4186-a0fc-0905443abacb@googlegroups.com>
Message-ID: <871sy1s4sn.fsf@handshake.de>

Alex <alxdoutsinis at gmail.com> writes:

> Can please anyone explaine me what do each of these code lines and how k-means algorithm works?

How about searching "wikipedia"?
(--> "https://en.wikipedia.org/wiki/K-means_clustering").



From marko at pacujo.net  Thu Nov 24 03:39:45 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Thu, 24 Nov 2016 10:39:45 +0200
Subject: Is this pythonic?
References: <o13meh$p2g$1@blaine.gmane.org>
 <mailman.48.1479892259.3146.python-list@python.org>
 <87inrer0dl.fsf@elektro.pacujo.net> <o13thn$b0m$1@blaine.gmane.org>
 <mailman.51.1479899527.3146.python-list@python.org>
 <583653bb$0$1603$c3e8da3$5496439d@news.astraweb.com>
 <o15rsm$vpp$1@blaine.gmane.org>
 <mailman.81.1479963365.3146.python-list@python.org>
 <58368358$0$1513$c3e8da3$5496439d@news.astraweb.com>
 <o1653p$65s$1@blaine.gmane.org>
 <mailman.83.1479972809.3146.python-list@python.org>
Message-ID: <871sy1qnji.fsf@elektro.pacujo.net>

"Frank Millman" <frank at chagford.com>:

> "Steven D'Aprano"  wrote in message
> news:58368358$0$1513$c3e8da3$5496439d at news.astraweb.com...

>> I'm a newbie to asyncio, but if I were doing this using threads, [...]
>
> To me, the beauty of asyncio (or I suppose async in general) is that I
> don't have to worry about any of what you describe above.

The programming model for threads and asyncio coroutines is identical.
The differences for the programmer are smallish details:

 * In asyncio, all functions that can potentially block must be tagged
   with "async" and all calls to such functions must be tagged with
   "await".

 * Not all blocking functions have an equivalent coroutine. All
   functions are readily available to threads.

 * Coroutines can be killed, threads cannot.

 * Coroutines can be multiplexed, threads cannot.


Marko


From __peter__ at web.de  Thu Nov 24 04:18:21 2016
From: __peter__ at web.de (Peter Otten)
Date: Thu, 24 Nov 2016 10:18:21 +0100
Subject: generating list of files matching condition
References: <87fumh4jhj.fsf@otaria.sebmel.org>
Message-ID: <o16b8q$a84$1@blaine.gmane.org>

Seb wrote:

> Hello,
> 
> Given a list of files:
> 
> In [81]: ec_files[0:10]
> Out[81]:
> 
> [u'EC_20160604002000.csv',
>  u'EC_20160604010000.csv',
>  u'EC_20160604012000.csv',
>  u'EC_20160604014000.csv',
>  u'EC_20160604020000.csv']
> 
> where the numbers are are a timestamp with format %Y%m%d%H%M%S, I'd like
> to generate a list of matching files for each 2-hr period in a 2-h
> frequency time series.  Ultimately I'm using Pandas to read and handle
> the data in each group of files.  For the task of generating the files
> for each 2-hr period, I've done the following:
> 
> beg_tstamp = pd.to_datetime(ec_files[0][-18:-4],
>                             format="%Y%m%d%H%M%S")
> end_tstamp = pd.to_datetime(ec_files[-1][-18:-4],
>                             format="%Y%m%d%H%M%S")
> tstamp_win = pd.date_range(beg_tstamp, end_tstamp, freq="2H")
> 
> So tstamp_win is the 2-hr frequency time series spanning the timestamps
> in the files in ec_files.
> 
> I've generated the list of matching files for each tstamp_win using a
> comprehension:
> 
> win_files = []
> for i, w in enumerate(tstamp_win):
>     nextw = w + pd.Timedelta(2, "h")
>     ifiles = [x for x in ec_files if
>               pd.to_datetime(x[-18:-4], format="%Y%m%d%H%M%S") >= w and
>               pd.to_datetime(x[-18:-4], format="%Y%m%d%H%M%S") < nextw]
>     win_files.append(ifiles)
> 
> However, this is proving very slow, and was wondering whether there's a
> better/faster way to do this.  Any tips would be appreciated.

Is win_files huge? Then it might help to avoid going over the entire list 
for every interval. Instead you can sort the list and then add to the 
current list while you are below nextw.

My pandas doesn't seem to have Timedelta (probably it's too old), so here's 
a generic solution using only the stdlib:

$ cat group_2hours.py
import itertools
import datetime
import pprint

def filename_to_time(filename):
    return datetime.datetime.strptime(filename[-18:-4], "%Y%m%d%H%M%S")

def make_key(delta_t):
    upper_bound = None
    def key(filename):
        nonlocal upper_bound

        if upper_bound is None:
            upper_bound = filename_to_time(filename) + delta_t
        else:
            t = filename_to_time(filename)
            while t >= upper_bound: # needs work if there are large gaps
                upper_bound += delta_t
        return upper_bound
    return key

ec_files = [
    u'EC_20160604002000.csv',
    u'EC_20160604010000.csv',
    u'EC_20160604012000.csv',
    u'EC_20160604014000.csv',
    u'EC_20160604020000.csv',
    u'EC_20160604050000.csv',
    u'EC_20160604060000.csv',
    u'EC_20160604070000.csv',
]
ec_files.sort() # ensure filenames are in ascending order

TWO_HOURS = datetime.timedelta(hours=2)

win_files = [
    list(group) for _key, group
    in itertools.groupby(ec_files, key=make_key(TWO_HOURS))
]

pprint.pprint(win_files)
$ python3 group_2hours.py 
[['EC_20160604002000.csv',
  'EC_20160604010000.csv',
  'EC_20160604012000.csv',
  'EC_20160604014000.csv',
  'EC_20160604020000.csv'],
 ['EC_20160604050000.csv', 'EC_20160604060000.csv'],
 ['EC_20160604070000.csv']]
$ 



PS: If the files' prefixes differ you cannot sort by name. Instead use

ec_files.sort(key=filename_to_time)

PPS: There is probably a way to do this by converting the list to a pandas 
dataframe; it might be worthwhile to ask in a specialised forum.



From rosuav at gmail.com  Thu Nov 24 04:58:19 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 24 Nov 2016 20:58:19 +1100
Subject: Is this pythonic?
In-Reply-To: <871sy1qnji.fsf@elektro.pacujo.net>
References: <o13meh$p2g$1@blaine.gmane.org>
 <mailman.48.1479892259.3146.python-list@python.org>
 <87inrer0dl.fsf@elektro.pacujo.net> <o13thn$b0m$1@blaine.gmane.org>
 <mailman.51.1479899527.3146.python-list@python.org>
 <583653bb$0$1603$c3e8da3$5496439d@news.astraweb.com>
 <o15rsm$vpp$1@blaine.gmane.org>
 <mailman.81.1479963365.3146.python-list@python.org>
 <58368358$0$1513$c3e8da3$5496439d@news.astraweb.com>
 <o1653p$65s$1@blaine.gmane.org>
 <mailman.83.1479972809.3146.python-list@python.org>
 <871sy1qnji.fsf@elektro.pacujo.net>
Message-ID: <CAPTjJmqsNUvYm4TiEfELQ6pBkE113CY6owxuVX5SPXOK61smuA@mail.gmail.com>

On Thu, Nov 24, 2016 at 7:39 PM, Marko Rauhamaa <marko at pacujo.net> wrote:
>  * Coroutines can be killed, threads cannot.
>

Not strictly true. A coroutine can be abandoned at an await point, but
the currently-executed call is still going to complete (usually); a
thread can be killed, but certain non-interruptible operations will
delay the termination until after that operation. So either way, the
operation still runs to completion.

ChrisA


From marko at pacujo.net  Thu Nov 24 05:59:48 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Thu, 24 Nov 2016 12:59:48 +0200
Subject: Is this pythonic?
References: <o13meh$p2g$1@blaine.gmane.org>
 <mailman.48.1479892259.3146.python-list@python.org>
 <87inrer0dl.fsf@elektro.pacujo.net> <o13thn$b0m$1@blaine.gmane.org>
 <mailman.51.1479899527.3146.python-list@python.org>
 <583653bb$0$1603$c3e8da3$5496439d@news.astraweb.com>
 <o15rsm$vpp$1@blaine.gmane.org>
 <mailman.81.1479963365.3146.python-list@python.org>
 <58368358$0$1513$c3e8da3$5496439d@news.astraweb.com>
 <o1653p$65s$1@blaine.gmane.org>
 <mailman.83.1479972809.3146.python-list@python.org>
 <871sy1qnji.fsf@elektro.pacujo.net>
 <CAPTjJmqsNUvYm4TiEfELQ6pBkE113CY6owxuVX5SPXOK61smuA@mail.gmail.com>
 <mailman.87.1479981508.3146.python-list@python.org>
Message-ID: <87wpftp2hn.fsf@elektro.pacujo.net>

Chris Angelico <rosuav at gmail.com>:

> On Thu, Nov 24, 2016 at 7:39 PM, Marko Rauhamaa <marko at pacujo.net> wrote:
>>  * Coroutines can be killed, threads cannot.
>
> Not strictly true. A coroutine can be abandoned at an await point, but
> the currently-executed call is still going to complete (usually);

I don't quite understand. Say you are awaiting on receiving bytes from a
socket. That means there has been a nonblocking call to read(2),
recvmsg(2) or equivalent that has returned EAGAIN. If you now abandon
the coroutine, there is no resumption of the system call but the
coroutine can finish instantaneously.

> a thread can be killed, but certain non-interruptible operations will
> delay the termination until after that operation. So either way, the
> operation still runs to completion.

There *could* be such noninterruptible operations (which would be a big
shame). They would have to be implemented with the help of a separate
thread. The surface coroutine can never sit in a blocking operation.


Marko


From rosuav at gmail.com  Thu Nov 24 06:06:13 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 24 Nov 2016 22:06:13 +1100
Subject: Is this pythonic?
In-Reply-To: <87wpftp2hn.fsf@elektro.pacujo.net>
References: <o13meh$p2g$1@blaine.gmane.org>
 <mailman.48.1479892259.3146.python-list@python.org>
 <87inrer0dl.fsf@elektro.pacujo.net> <o13thn$b0m$1@blaine.gmane.org>
 <mailman.51.1479899527.3146.python-list@python.org>
 <583653bb$0$1603$c3e8da3$5496439d@news.astraweb.com>
 <o15rsm$vpp$1@blaine.gmane.org>
 <mailman.81.1479963365.3146.python-list@python.org>
 <58368358$0$1513$c3e8da3$5496439d@news.astraweb.com>
 <o1653p$65s$1@blaine.gmane.org>
 <mailman.83.1479972809.3146.python-list@python.org>
 <871sy1qnji.fsf@elektro.pacujo.net>
 <CAPTjJmqsNUvYm4TiEfELQ6pBkE113CY6owxuVX5SPXOK61smuA@mail.gmail.com>
 <mailman.87.1479981508.3146.python-list@python.org>
 <87wpftp2hn.fsf@elektro.pacujo.net>
Message-ID: <CAPTjJmr=Pw1BotiKLEFL6YErPmy4D+aGnUTpyxBYd68EBm6amA@mail.gmail.com>

On Thu, Nov 24, 2016 at 9:59 PM, Marko Rauhamaa <marko at pacujo.net> wrote:
> Chris Angelico <rosuav at gmail.com>:
>
>> On Thu, Nov 24, 2016 at 7:39 PM, Marko Rauhamaa <marko at pacujo.net> wrote:
>>>  * Coroutines can be killed, threads cannot.
>>
>> Not strictly true. A coroutine can be abandoned at an await point, but
>> the currently-executed call is still going to complete (usually);
>
> I don't quite understand. Say you are awaiting on receiving bytes from a
> socket. That means there has been a nonblocking call to read(2),
> recvmsg(2) or equivalent that has returned EAGAIN. If you now abandon
> the coroutine, there is no resumption of the system call but the
> coroutine can finish instantaneously.

Is the read(2) still going to consume data from the pipe/socket? If
so, the operation is still going to continue, whether you use
coroutines or threads. If not, it would have been cancelled whether
you use coroutines or threads.

ChrisA


From marko at pacujo.net  Thu Nov 24 06:14:28 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Thu, 24 Nov 2016 13:14:28 +0200
Subject: Is this pythonic?
References: <o13meh$p2g$1@blaine.gmane.org>
 <mailman.48.1479892259.3146.python-list@python.org>
 <87inrer0dl.fsf@elektro.pacujo.net> <o13thn$b0m$1@blaine.gmane.org>
 <mailman.51.1479899527.3146.python-list@python.org>
 <583653bb$0$1603$c3e8da3$5496439d@news.astraweb.com>
 <o15rsm$vpp$1@blaine.gmane.org>
 <mailman.81.1479963365.3146.python-list@python.org>
 <58368358$0$1513$c3e8da3$5496439d@news.astraweb.com>
 <o1653p$65s$1@blaine.gmane.org>
 <mailman.83.1479972809.3146.python-list@python.org>
 <871sy1qnji.fsf@elektro.pacujo.net>
 <CAPTjJmqsNUvYm4TiEfELQ6pBkE113CY6owxuVX5SPXOK61smuA@mail.gmail.com>
 <mailman.87.1479981508.3146.python-list@python.org>
 <87wpftp2hn.fsf@elektro.pacujo.net>
 <CAPTjJmr=Pw1BotiKLEFL6YErPmy4D+aGnUTpyxBYd68EBm6amA@mail.gmail.com>
 <mailman.88.1479985576.3146.python-list@python.org>
Message-ID: <87shqhp1t7.fsf@elektro.pacujo.net>

Chris Angelico <rosuav at gmail.com>:

> On Thu, Nov 24, 2016 at 9:59 PM, Marko Rauhamaa <marko at pacujo.net> wrote:
>> Chris Angelico <rosuav at gmail.com>:
>>> A coroutine can be abandoned at an await point, but the
>>> currently-executed call is still going to complete (usually);
>>
>> I don't quite understand. Say you are awaiting on receiving bytes from a
>> socket. That means there has been a nonblocking call to read(2),
>> recvmsg(2) or equivalent that has returned EAGAIN. If you now abandon
>> the coroutine, there is no resumption of the system call but the
>> coroutine can finish instantaneously.
>
> Is the read(2) still going to consume data from the pipe/socket?

Only if the kernel has already buffered data it has received previously.

If there isn't yet any data available in the kernel, read(2) returns
with errno=EAGAIN, and the control is returned to the main loop. The
main loop then goes to sleep in epoll_wait(2) or equivalent.

> If so, the operation is still going to continue, whether you use
> coroutines or threads. If not, it would have been cancelled whether
> you use coroutines or threads.

When you use threads, you call read(2) in the blocking mode. Then the
read(2) operation will block "for ever." There's no clean way to cancel
the system call.


Marko


From rosuav at gmail.com  Thu Nov 24 06:38:43 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 24 Nov 2016 22:38:43 +1100
Subject: Is this pythonic?
In-Reply-To: <87shqhp1t7.fsf@elektro.pacujo.net>
References: <o13meh$p2g$1@blaine.gmane.org>
 <mailman.48.1479892259.3146.python-list@python.org>
 <87inrer0dl.fsf@elektro.pacujo.net> <o13thn$b0m$1@blaine.gmane.org>
 <mailman.51.1479899527.3146.python-list@python.org>
 <583653bb$0$1603$c3e8da3$5496439d@news.astraweb.com>
 <o15rsm$vpp$1@blaine.gmane.org>
 <mailman.81.1479963365.3146.python-list@python.org>
 <58368358$0$1513$c3e8da3$5496439d@news.astraweb.com>
 <o1653p$65s$1@blaine.gmane.org>
 <mailman.83.1479972809.3146.python-list@python.org>
 <871sy1qnji.fsf@elektro.pacujo.net>
 <CAPTjJmqsNUvYm4TiEfELQ6pBkE113CY6owxuVX5SPXOK61smuA@mail.gmail.com>
 <mailman.87.1479981508.3146.python-list@python.org>
 <87wpftp2hn.fsf@elektro.pacujo.net>
 <CAPTjJmr=Pw1BotiKLEFL6YErPmy4D+aGnUTpyxBYd68EBm6amA@mail.gmail.com>
 <mailman.88.1479985576.3146.python-list@python.org>
 <87shqhp1t7.fsf@elektro.pacujo.net>
Message-ID: <CAPTjJmoNWTeDSQtr8QNU1SpDt0Q29G8TUX+1namG2jbzzuHgEg@mail.gmail.com>

On Thu, Nov 24, 2016 at 10:14 PM, Marko Rauhamaa <marko at pacujo.net> wrote:
> When you use threads, you call read(2) in the blocking mode. Then the
> read(2) operation will block "for ever." There's no clean way to cancel
> the system call.

Signals will usually interrupt system calls, causing them to return
EINTR. There are exceptions (the aforementioned uninterruptible calls,
but they're not available in nonblocking form, so they're the same for
threads and coroutines), but the bulk of system calls will halt
cleanly on receipt of a signal. And yes, you CAN send signals to
specific threads; there are limitations, but for a language like
Python, there's no difficulty in having a single disposition for (say)
SIGINT, and then using thread signalling to figure out which thread
should have KeyboardInterrupt raised in it.

ChrisA


From twgrops at googlemail.com  Thu Nov 24 07:42:06 2016
From: twgrops at googlemail.com (Thomas Grops)
Date: Thu, 24 Nov 2016 04:42:06 -0800 (PST)
Subject: Help with two issues, buttons and second class object
Message-ID: <f44a5197-0c3d-4a19-aad9-983244c8aa26@googlegroups.com>

Hi I have created some code, which moves a rectangle around and when it hits the edge it picks a random new direction. It does this by the count function within my class. I am wanting to create a button to randomly change count but I my class seems to be getting errors.

I also wanted to create a second class object tank2 but it doesn't seem to move when I create it with tank2.Tank(x,y,vx,vy) and call tank1.move()

Can anyone help me many thanks.

from tkinter import *
import time
import random

#Properties for the tank
class Tank():
    
    

    #life, speed, starting position, vectors, size of tank
    def __init__(self, x, y, vx, vy):
        self.__life=10
        #self.canvas=canvas
        self.speed = 1
        self.x = x
        self.y = y
        self.vx = vx
        self.vy = vy
        self.count=0
        #self.w = tank.width()
        #self.h = tank.height()
        self.id = canvas.create_rectangle(x,y,vx,vy, fill='green')
        #Create left button and call function to print left
        '''buttonL=Button(canvas,text='Change Direction', command=self.changeDirection())
        buttonL.pack(side=BOTTOM, anchor=SW)'''
        #print(self.x)
    #tank attacked
    def attack(self):
        print('ouch!')
        self.__life -= 1

    #check life
    def checkLife(self):
        if self.__life <= 0:
            print('dead')
        else:
            print(str(self.__life) + " life left")

    #respawn at starting point
    #def respawn(self):
    

    #medic pack
    def medic(self):
        self.__life += 5

    #move directions
    def right(self):
        canvas.move(self.id,+5,0)#move right
        
        #reposition x,vx,y,vy values
        self.x+=5
        self.vx+=5

        #Update canvas
        canvas.update()
        time.sleep(0.1)
        
    def left(self):
        canvas.move(self.id,-5,0)#move left
        
        #reposition x,vx,y,vy values
        self.x+=-5
        self.vx+=-5

        #Update canvas
        canvas.update()
        time.sleep(0.1)

    def up(self):
        canvas.move(self.id,0,-2)#move up
        
        #reposition x,vx,y,vy values
        self.y+=-2
        self.vy+=-2

        #Update canvas
        canvas.update()
        time.sleep(0.1)

    def down(self):
        canvas.move(self.id,0,+2)#move down
        
        #reposition x,vx,y,vy values
        self.y+=2
        self.vy+=2

        #Update canvas
        canvas.update()
        time.sleep(0.1)

    def upLeft(self):
        canvas.move(self.id,-1,-1)#move upLeft
        
        #reposition x,vx,y,vy values
        self.y+=-1
        self.vy+=-1
        self.x+=-1
        self.vx+=-1

        #Update canvas
        canvas.update()
        time.sleep(0.1)

    def upRight(self):
        canvas.move(self.id,+1,-1)#move upRight
        
        #reposition x,vx,y,vy values
        self.y+=-1
        self.vx+=1
        self.vy+=-1
        self.x+=1
        
        #Update canvas
        canvas.update()
        time.sleep(0.1)

    def downLeft(self):
        canvas.move(self.id,-1,+1)#move downLeft

        #reposition x,vx,y,vy values
        self.x+=-1
        self.vx+=-1
        self.y+=1
        self.vy+=1
        
        #Update canvas
        canvas.update()
        time.sleep(0.1)

    def downRight(self):
        #move downRight
        canvas.move(self.id,+1,+1)

        #reposition x,vx,y,vy values
        self.x+=1
        self.vx+=1
        self.y+=1
        self.vy+=1
        
        #Update canvas
        canvas.update()
        time.sleep(0.1)

    

    def count(self,count):
        #Count triggers direction of movement
        self.count = count
        print (count)



    #movement
    def move(self):
        
        # Loop for steps in movement
        for t in range(1, 10000):
            #Move direction depending on count value
            if self.count==0:
                self.left() 
            if self.count==1:
                self.right()
            if self.count==2:
                self.up()
            if self.count==3:
                self.down()
            if self.count==4:
                self.upLeft()
            if self.count==5:
                self.upRight()
            if self.count==6:
                self.downRight()
            if self.count==7:
                self.downLeft()

         # If a boundary has been crossed, pick a direction randomly

            #Left border
            if self.x <= 0:
                #banned directions
                excludedNumbers = [0,4,7]
                #define random integer to be selected
                randomNumber = random.randint(0,8)
                #nested while loop so that the banned directions are not selected
                while randomNumber in excludedNumbers:
                    randomNumber = random.randint(0,8)
                #feed allowed random direction back to the count
                self.count=randomNumber

            #Right border
            elif self.vx >= 1000:
                #banned directions
                excludedNumbers = [1,5,6]
                #define random integer to be selected
                randomNumber = random.randint(0,8)
                #nested while loop so that the banned directions are not selected
                while randomNumber in excludedNumbers:
                    randomNumber = random.randint(0,8)
                #feed allowed random direction back to the count
                self.count=randomNumber
    
            #Top border
            elif self.y <= 0:
                #banned directions
                excludedNumbers = [2,4,5]
                #define random integer to be selected
                randomNumber = random.randint(0,8)
                #nested while loop so that the banned directions are not selected
                while randomNumber in excludedNumbers:
                    randomNumber = random.randint(0,8)
                #feed allowed random direction back to the count
                self.count=randomNumber

            #Bottom border
            elif self.vy >= 700:
                #banned directions
                excludedNumbers = [3,6,7]
                #define random integer to be selected
                randomNumber = random.randint(0,8)
                #nested while loop so that the banned directions are not selected
                while randomNumber in excludedNumbers:
                    randomNumber = random.randint(0,8)
                #feed allowed random direction back to the count
                self.count=randomNumber

    '''def changeDirection():
            randomNumber=random.randint(0,8)
            self.count=randomNumber'''



'class Wall(object):'
    
#Main=======================================================================================================
main = Tk() 

#Canvas size
WIDTH=1000
HEIGHT=700
    
    
'''def changeDirection():
    test=Tank()
    test.count()
    randomNumber=random.randint(0,8)
    tank1.count=randomNumber'''

'''# The velocity, or distance moved per time step
vx = 10.0    # x velocity
vy = 5.0    # y velocity

# Boundaries
x_min = 0.0
y_min = 0.0
x_max = HEIGHT
y_max = WIDTH'''
    
#Show canvas
canvas = Canvas(main,width=WIDTH, height=HEIGHT, bg='khaki')

canvas.pack()

#Create a tank
tank1= Tank(950,650,950+40,650+40)


#testing features
'''tank1.attack()
tank1.attack()
tank1.checkLife()
tank1.medic()
tank1.checkLife()
tank1.move()'''

tank1.move()

   
      
canvas.pack(padx=10,pady=10)

#Complete the GUI
main.mainloop()


From marko at pacujo.net  Thu Nov 24 08:10:06 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Thu, 24 Nov 2016 15:10:06 +0200
Subject: Is this pythonic?
References: <o13meh$p2g$1@blaine.gmane.org>
 <mailman.48.1479892259.3146.python-list@python.org>
 <87inrer0dl.fsf@elektro.pacujo.net> <o13thn$b0m$1@blaine.gmane.org>
 <mailman.51.1479899527.3146.python-list@python.org>
 <583653bb$0$1603$c3e8da3$5496439d@news.astraweb.com>
 <o15rsm$vpp$1@blaine.gmane.org>
 <mailman.81.1479963365.3146.python-list@python.org>
 <58368358$0$1513$c3e8da3$5496439d@news.astraweb.com>
 <o1653p$65s$1@blaine.gmane.org>
 <mailman.83.1479972809.3146.python-list@python.org>
 <871sy1qnji.fsf@elektro.pacujo.net>
 <CAPTjJmqsNUvYm4TiEfELQ6pBkE113CY6owxuVX5SPXOK61smuA@mail.gmail.com>
 <mailman.87.1479981508.3146.python-list@python.org>
 <87wpftp2hn.fsf@elektro.pacujo.net>
 <CAPTjJmr=Pw1BotiKLEFL6YErPmy4D+aGnUTpyxBYd68EBm6amA@mail.gmail.com>
 <mailman.88.1479985576.3146.python-list@python.org>
 <87shqhp1t7.fsf@elektro.pacujo.net>
 <CAPTjJmoNWTeDSQtr8QNU1SpDt0Q29G8TUX+1namG2jbzzuHgEg@mail.gmail.com>
 <mailman.89.1479987526.3146.python-list@python.org>
Message-ID: <87oa15owgh.fsf@elektro.pacujo.net>

Chris Angelico <rosuav at gmail.com>:

> On Thu, Nov 24, 2016 at 10:14 PM, Marko Rauhamaa <marko at pacujo.net> wrote:
>> When you use threads, you call read(2) in the blocking mode. Then the
>> read(2) operation will block "for ever." There's no clean way to
>> cancel the system call.
>
> Signals will usually interrupt system calls, causing them to return
> EINTR. There are exceptions (the aforementioned uninterruptible calls,
> but they're not available in nonblocking form, so they're the same for
> threads and coroutines), but the bulk of system calls will halt
> cleanly on receipt of a signal. And yes, you CAN send signals to
> specific threads; there are limitations, but for a language like
> Python, there's no difficulty in having a single disposition for (say)
> SIGINT, and then using thread signalling to figure out which thread
> should have KeyboardInterrupt raised in it.

Yes, pthread_kill(3) has been made available in Python-3.3, I'm
noticing.

Also:

   Changed in version 3.5: Python now retries system calls when a
   syscall is interrupted by a signal, except if the signal handler
   raises an exception (see PEP 475 for the rationale), instead of
   raising InterruptedError.

   <URL: https://docs.python.org/3/library/exceptions.html#Interrupt
   edError>


Marko


From gengyangcai at gmail.com  Thu Nov 24 09:00:20 2016
From: gengyangcai at gmail.com (Cai Gengyang)
Date: Thu, 24 Nov 2016 06:00:20 -0800 (PST)
Subject: NameError
In-Reply-To: <mailman.82.1479964961.3146.python-list@python.org>
References: <223a00e3-1e8b-4ba3-9402-37a7f8ba9dcf@googlegroups.com>
 <CAPM-O+ySA-o3Afh12NU6Ce0REC_pnoH7_we4AOE-NYe9pSqOTw@mail.gmail.com>
 <mailman.77.1479956046.3146.python-list@python.org>
 <079079bb-6abd-4660-be25-dd4f33165e35@googlegroups.com>
 <33bcdbba-9c5d-d884-366b-5897812afbd0@gmx.com>
 <mailman.80.1479960005.3146.python-list@python.org>
 <e4c9f90e-3ef0-41b1-b729-a48792ebada9@googlegroups.com>
 <CAPjBxg9LrwO3t2VVEs9p+fTL8a6HuCLbyF3rbMAqOU+e=0wNGQ@mail.gmail.com>
 <mailman.82.1479964961.3146.python-list@python.org>
Message-ID: <a6a76fae-2371-40f7-abed-186a6f1fc023@googlegroups.com>

CaiGengYangs-MacBook-Pro:~ CaiGengYang$ import pygame
-bash: import: command not found



On Thursday, November 24, 2016 at 1:22:56 PM UTC+8, Nathan Ernst wrote:
> I don't see anything in that output resembling an error, just a few
> warnings that some features may no be available.
> 
> Have you tried importing pygame after you did that? That's what'll prove
> one way or another that it worked.
> 
> Regards,
> Nate
> 
> On Wed, Nov 23, 2016 at 10:48 PM, Cai Gengyang <gengyangcai at gmail.com>
> wrote:
> 
> > Yea, using Mac
> >
> > Following the instructions here for Mac ---https://bitbucket.org/
> > pygame/pygame/issues/82/homebrew-on-leopard-fails-to-
> > install#comment-627494
> >
> > GengYang Cai CaiGengYangs-MacBook-Pro:~ CaiGengYang$ brew install python
> > ==> Installing dependencies for python: xz, pkg-config, readline, sqlite,
> > ==> Installing python dependency: xz
> > ==> Downloading https://homebrew.bintray.com/.
> > ../xz-5.2.2.yosemite.bottle.ta
> > ########################################################################
> > 100.0%
> > ==> Pouring xz-5.2.2.yosemite.bottle.tar.gz
> > ? /usr/local/Cellar/xz/5.2.2: 91 files, 1.4M
> > ==> Installing python dependency: pkg-config
> > ==> Downloading https://homebrew.bintray.com/.
> > ../pkg-config-0.29.1_1.yosemit
> > ########################################################################
> > 100.0%
> > ==> Pouring pkg-config-0.29.1_1.yosemite.bottle.tar.gz
> > ? /usr/local/Cellar/pkg-config/0.29.1_1: 10 files, 627.3K
> > ==> Installing python dependency: readline
> > ==> Downloading https://homebrew.bintray.com/.
> > ../readline-6.3.8.yosemite.bot
> > ########################################################################
> > 100.0%
> > ==> Pouring readline-6.3.8.yosemite.bottle.tar.gz
> > ==> Caveats
> > This formula is keg-only, which means it was not symlinked into /usr/local.
> >
> > OS X provides the BSD libedit library, which shadows libreadline.
> > In order to prevent conflicts when programs look for libreadline we are
> > defaulting this GNU Readline installation to keg-only.
> >
> > Generally there are no consequences of this for you. If you build your
> > own software and it requires this formula, you'll need to add to your
> > build variables:
> >
> > LDFLAGS: -L/usr/local/opt/readline/lib
> > CPPFLAGS: -I/usr/local/opt/readline/include
> >
> > ==> Summary
> > ? /usr/local/Cellar/readline/6.3.8: 46 files, 2M
> > ==> Installing python dependency: sqlite
> > ==> Downloading https://homebrew.bintray.com/.
> > ../sqlite-3.13.0.yosemite.bott
> > ########################################################################
> > 100.0%
> > ==> Pouring sqlite-3.13.0.yosemite.bottle.tar.gz
> > ==> Caveats
> > This formula is keg-only, which means it was not symlinked into /usr/local.
> >
> > OS X provides an older sqlite3.
> >
> > Generally there are no consequences of this for you. If you build your
> > own software and it requires this formula, you'll need to add to your
> > build variables:
> >
> > LDFLAGS: -L/usr/local/opt/sqlite/lib
> > CPPFLAGS: -I/usr/local/opt/sqlite/include
> >
> > ==> Summary
> > ? /usr/local/Cellar/sqlite/3.13.0: 10 files, 2.9M
> > ==> Installing python dependency: gdbm
> > ==> Downloading https://homebrew.bintray.com/.
> > ../gdbm-1.12.yosemite.bottle.t
> > ########################################################################
> > 100.0%
> > ==> Pouring gdbm-1.12.yosemite.bottle.tar.gz
> > ? /usr/local/Cellar/gdbm/1.12: 18 files, 490.8K
> > ==> Installing python dependency: openssl
> > ==> Downloading https://homebrew.bintray.com/.
> > ../openssl-1.0.2h_1.yosemite.b
> > ########################################################################
> > 100.0%
> > ==> Pouring openssl-1.0.2h_1.yosemite.bottle.tar.gz
> > ==> Caveats
> > A CA file has been bootstrapped using certificates from the system
> > keychain. To add additional certificates, place .pem files in
> > /usr/local/etc/openssl/certs
> >
> > and run
> > /usr/local/opt/openssl/bin/c_rehash
> >
> > This formula is keg-only, which means it was not symlinked into /usr/local.
> >
> > Apple has deprecated use of OpenSSL in favor of its own TLS and crypto
> > libraries
> >
> > Generally there are no consequences of this for you. If you build your
> > own software and it requires this formula, you'll need to add to your
> > build variables:
> >
> > LDFLAGS: -L/usr/local/opt/openssl/lib
> > CPPFLAGS: -I/usr/local/opt/openssl/include
> >
> > ==> Summary
> > ? /usr/local/Cellar/openssl/1.0.2h_1: 1,691 files, 12.0M
> > ==> Installing python
> > Warning: Building python from source:
> > The bottle needs the Apple Command Line Tools to be installed.
> > You can install them, if desired, with:
> > xcode-select --install
> >
> > ==> Downloading https://www.python.org/.../2.7.12/Python-2.7.12.tar.xz
> > ########################################################################
> > 100.0%
> > ==> Downloading https://bugs.python.org/file30805/issue10910-
> > workaround.txt
> > ########################################################################
> > 100.0%
> > ==> Patching
> > ==> Applying issue10910-workaround.txt
> > patching file Include/pyport.h
> > Hunk #1 succeeded at 713 (offset 14 lines).
> > Hunk #2 succeeded at 736 (offset 14 lines).
> > ==> ./configure --prefix=/usr/local/Cellar/python/2.7.12 --enable-ipv6
> > --dataroo
> > ==> make
> > /usr/local/share/python/easy_install mecurial
> >
> > brew install sdl
> > brew install sdl_mixer
> > brew install sdl_ttf
> > brew install sdl_image
> >
> > hg clone https://bitbucket.org/pygame/pygame
> > cd pygame
> > /usr/local/bin/python setup.py install
> >
> >
> > Does this work ?
> >
> >
> >
> >
> >
> >
> >
> > On Thursday, November 24, 2016 at 12:00:18 PM UTC+8, Thomas Nyberg wrote:
> > > On 11/23/2016 10:02 PM, Cai Gengyang wrote:
> > > > I tried to import pygame by using these commands
> > ------------------------------------------------https://www.
> > google.com.sg/#q=how+to+import+pygame
> > > >
> > > > but this is the error I got :
> > > >
> > > > CaiGengYangs-MacBook-Pro:~ CaiGengYang$ sudo apt-get install
> > python-pygame
> > > > sudo: apt-get: command not found
> > > >
> > > Judging by your prompt it looks like you're on a Mac. The apt-get
> > > program is installed on some Linux distributions (probably your
> > > instructions are for Ubuntu). Here are some istructions for installing
> > > pygame for a Macbook:
> > >
> > >       http://pygame.org/wiki/macintosh
> > >       http://florian-berger.de/en/articles/installing-pygame-
> > for-python-3-on-os-x/
> > >       https://jamesfriend.com.au/installing-pygame-python-3-
> > mac-os-yosemite
> > >
> > > Here's some info about getting a package manager like apt for Mac:
> > >
> > >       http://unix.stackexchange.com/questions/80711/how-to-
> > install-apt-get-or-yum-on-mac-os-x
> > >
> > > Hopefully this helps.
> > >
> > > Cheers,
> > > Thomas
> > --
> > https://mail.python.org/mailman/listinfo/python-list
> >


From __peter__ at web.de  Thu Nov 24 09:53:26 2016
From: __peter__ at web.de (Peter Otten)
Date: Thu, 24 Nov 2016 15:53:26 +0100
Subject: Help with two issues, buttons and second class object
References: <f44a5197-0c3d-4a19-aad9-983244c8aa26@googlegroups.com>
Message-ID: <o16ut3$78m$1@blaine.gmane.org>

Thomas Grops via Python-list wrote:

> Hi I have created some code, which moves a rectangle around and when it
> hits the edge it picks a random new direction. It does this by the count
> function within my class. I am wanting to create a button to randomly
> change count but I my class seems to be getting errors.
> 
> I also wanted to create a second class object tank2 but it doesn't seem to
> move when I create it with tank2.Tank(x,y,vx,vy) and call tank1.move()
> 
> Can anyone help me many thanks.

It may take some time to get it, but don't use time.sleep() in a tkinter 
script!

>     def right(self):
>         canvas.move(self.id,+5,0)#move right
>         
>         #reposition x,vx,y,vy values
>         self.x+=5
>         self.vx+=5
> 
>         #Update canvas
>         canvas.update()
>         time.sleep(0.1)

When you have to repeat the same code with small variations (the 
canvas.move() arguments) make these variations arguments of the function.

>     def move(self):
>         
>         # Loop for steps in movement
>         for t in range(1, 10000):
>             #Move direction depending on count value
>             if self.count==0:
>                 self.left()

So 0 means "left" and 6 means "down and right". If a value's meaning is not 
obvious use a name instead

LEFT = 42 # also have a look at the enum module in the stdlib

if self.count == LEFT:
    self.left()

>             if self.count==1:
>                 self.right()
>             if self.count==2:
>                 self.up()
>             if self.count==3:
>                 self.down()
>             if self.count==4:
>                 self.upLeft()
>             if self.count==5:
>                 self.upRight()
>             if self.count==6:
>                 self.downRight()
>             if self.count==7:
>                 self.downLeft()

That's quite a lot of if-s. It might be better to use a table.

>             #Left border
>             if self.x <= 0:
>                 #banned directions
>                 excludedNumbers = [0,4,7]
>                 #define random integer to be selected
>                 randomNumber = random.randint(0,8)
>                 #nested while loop so that the banned directions are not
>                 #selected
>                 while randomNumber in excludedNumbers:
>                     randomNumber = random.randint(0,8)
>                 #feed allowed random direction back to the count
>                 self.count=randomNumber
> 
>             #Right border
>             elif self.vx >= 1000:
>                 #banned directions
>                 excludedNumbers = [1,5,6]
>                 #define random integer to be selected
>                 randomNumber = random.randint(0,8)
>                 #nested while loop so that the banned directions are not
>                 #selected
>                 while randomNumber in excludedNumbers:
>                     randomNumber = random.randint(0,8)
>                 #feed allowed random direction back to the count

Instead of repeating your code with copy-and-past make a helper function 
like the randx() posted by Larry Hudson.

By the way, randint(min, max) may return max so there are 9 possible 
outcomes while you handle only 8. To be consistent with Python's half-open 
ranges use 

random.randrange(8) # may return 0, 1, 2, 3, 4, 5, 6, 7, but not 8

Below is what became of your code when I messed with it. Don't copy it, try 
to pick up ideas, and have fun!

import tkinter as tk
import random

#Canvas size
WIDTH = 1000
HEIGHT = 700

BUTTONS_PER_ROW = 6

MRIGHT = 0
MLEFT = 1
MUP = 2
MDOWN = 3
MLEFTUP = 4
MRIGHTUP = 5
MRIGHTDOWN = 6
MLEFTDOWN = 7


class Tank():
    def __init__(self, root, canvas, name, x, y, width, height, color):
        self.root = root
        self.canvas = canvas
        self.color = color
        self.name = name
        self.__life = 10
        self.speed = 1
        self.x = x
        self.y = y
        self.width = width
        self.height = height
        self.direction = MRIGHT
        self.id = canvas.create_rectangle(
            x, y, self.right, self.bottom, fill=color
        )
        self.moves = [
            (5, 0),    # right
            (-5, 0),   # left
            (0, -2),   # up
            (0, 2),    # down
            (-1, -1),  # left-up
            (1, -1),   # right-up
            (1, 1),    # right-down
            (-1, 1)    # left-down
        ]

    @property
    def right(self):
        return self.x + self.width

    @property
    def bottom(self):
        return self.y + self.height

    def attack(self):
        print('ouch!')
        self.__life -= 1

    def checkLife(self):
        if self.__life <= 0:
            print('dead')
        else:
            print(str(self.__life) + " life left")

    def medic(self):
        self.__life += 5

    def move_tank(self, dx, dy):
        self.x += dx
        self.y += dy
        self.canvas.move(self.id, dx, dy)

    def move(self):
        self.move_tank(*self.moves[self.direction])

        # If a boundary has been crossed, pick a direction randomly
        directions = set(range(8))
        if self.x <= 0:
            directions -= {MLEFT, MLEFTUP, MLEFTDOWN}
        elif self.right >= WIDTH:
            directions -= {MRIGHT, MRIGHTUP, MRIGHTDOWN}

        if self.y <= 0:
            directions -= {MUP, MLEFTUP, MRIGHTUP}
        elif self.bottom >= HEIGHT:
            directions -= {MDOWN, MLEFTDOWN, MRIGHTDOWN}
        assert directions, "nowhere to go!"

        if len(directions) < 8:
            print(self, "hit wall, changing direction")
            self.direction = random.choice(list(directions))

        self.root.after(10, self.move)

    def changeDirection(self):
        print(self, "changing direction on user request")
        self.direction = random.randrange(8)

    def __str__(self):
        return self.name


def random_color():
    colors = [
        random.randrange(128),
        random.randrange(128, 256),
        random.randrange(128)
    ]
    random.shuffle(colors)
    return "#{:02x}{:02x}{:02x}".format(*colors)


def main():
    root = tk.Tk()

    canvas = tk.Canvas(root, width=WIDTH, height=HEIGHT, bg='khaki')
    canvas.grid(row=0, column=0, columnspan=BUTTONS_PER_ROW)

    tanks = [
        Tank(
            root, canvas, "Tank #{}".format(i),
            x, 650, 40, 40,
            random_color())
        for i, x in enumerate(range(0, WIDTH, 80), 1)
    ]
    for i, tank in enumerate(tanks):
        tank.move()
        button = tk.Button(
            root,
            text=tank.name,
            command=tank.changeDirection,
            bg=tank.color
            )
        button.grid(row=1+i // BUTTONS_PER_ROW, column=i % BUTTONS_PER_ROW)

    root.mainloop()

if __name__ == "__main__":
    main()

PS:

>     #life, speed, starting position, vectors, size of tank
>     def __init__(self, x, y, vx, vy):

Python has docstrings. Use them!




From tomuxiong at gmx.com  Thu Nov 24 09:59:17 2016
From: tomuxiong at gmx.com (Thomas Nyberg)
Date: Thu, 24 Nov 2016 09:59:17 -0500
Subject: NameError
In-Reply-To: <a6a76fae-2371-40f7-abed-186a6f1fc023@googlegroups.com>
References: <223a00e3-1e8b-4ba3-9402-37a7f8ba9dcf@googlegroups.com>
 <CAPM-O+ySA-o3Afh12NU6Ce0REC_pnoH7_we4AOE-NYe9pSqOTw@mail.gmail.com>
 <mailman.77.1479956046.3146.python-list@python.org>
 <079079bb-6abd-4660-be25-dd4f33165e35@googlegroups.com>
 <33bcdbba-9c5d-d884-366b-5897812afbd0@gmx.com>
 <mailman.80.1479960005.3146.python-list@python.org>
 <e4c9f90e-3ef0-41b1-b729-a48792ebada9@googlegroups.com>
 <CAPjBxg9LrwO3t2VVEs9p+fTL8a6HuCLbyF3rbMAqOU+e=0wNGQ@mail.gmail.com>
 <mailman.82.1479964961.3146.python-list@python.org>
 <a6a76fae-2371-40f7-abed-186a6f1fc023@googlegroups.com>
Message-ID: <b30df68c-7e71-2a78-21aa-4bf3e505d8bb@gmx.com>

On 11/24/2016 09:00 AM, Cai Gengyang wrote:
> CaiGengYangs-MacBook-Pro:~ CaiGengYang$ import pygame
> -bash: import: command not found

That indicates you're running "import pygame" at the bash interpreter 
prompt. You should first start "python" (type the word python without 
quotes and press enter) which starts the python interpreter. You should 
see this: ">>>". After this, then type "import pygame". See if that works.

Cheers,
Thomas



From gengyangcai at gmail.com  Thu Nov 24 10:05:17 2016
From: gengyangcai at gmail.com (Cai Gengyang)
Date: Thu, 24 Nov 2016 07:05:17 -0800 (PST)
Subject: NameError
In-Reply-To: <mailman.91.1479999510.3146.python-list@python.org>
References: <223a00e3-1e8b-4ba3-9402-37a7f8ba9dcf@googlegroups.com>
 <CAPM-O+ySA-o3Afh12NU6Ce0REC_pnoH7_we4AOE-NYe9pSqOTw@mail.gmail.com>
 <mailman.77.1479956046.3146.python-list@python.org>
 <079079bb-6abd-4660-be25-dd4f33165e35@googlegroups.com>
 <33bcdbba-9c5d-d884-366b-5897812afbd0@gmx.com>
 <mailman.80.1479960005.3146.python-list@python.org>
 <e4c9f90e-3ef0-41b1-b729-a48792ebada9@googlegroups.com>
 <CAPjBxg9LrwO3t2VVEs9p+fTL8a6HuCLbyF3rbMAqOU+e=0wNGQ@mail.gmail.com>
 <mailman.82.1479964961.3146.python-list@python.org>
 <a6a76fae-2371-40f7-abed-186a6f1fc023@googlegroups.com>
 <b30df68c-7e71-2a78-21aa-4bf3e505d8bb@gmx.com>
 <mailman.91.1479999510.3146.python-list@python.org>
Message-ID: <600d3d71-4b38-4196-b5b2-6c96aa7cfbce@googlegroups.com>

CaiGengYangs-MacBook-Pro:~ CaiGengYang$ python
Python 2.7.10 (v2.7.10:15c95b7d81dc, May 23 2015, 09:33:12) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pygame
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named pygame
>>> "import pygame"
'import pygame'
>>> 











On Thursday, November 24, 2016 at 10:58:41 PM UTC+8, Thomas Nyberg wrote:
> On 11/24/2016 09:00 AM, Cai Gengyang wrote:
> > CaiGengYangs-MacBook-Pro:~ CaiGengYang$ import pygame
> > -bash: import: command not found
> 
> That indicates you're running "import pygame" at the bash interpreter 
> prompt. You should first start "python" (type the word python without 
> quotes and press enter) which starts the python interpreter. You should 
> see this: ">>>". After this, then type "import pygame". See if that works.
> 
> Cheers,
> Thomas



From alister.ware at ntlworld.com  Thu Nov 24 10:06:52 2016
From: alister.ware at ntlworld.com (alister)
Date: Thu, 24 Nov 2016 15:06:52 GMT
Subject: NameError
References: <223a00e3-1e8b-4ba3-9402-37a7f8ba9dcf@googlegroups.com>
 <CAPM-O+ySA-o3Afh12NU6Ce0REC_pnoH7_we4AOE-NYe9pSqOTw@mail.gmail.com>
 <mailman.77.1479956046.3146.python-list@python.org>
 <079079bb-6abd-4660-be25-dd4f33165e35@googlegroups.com>
 <33bcdbba-9c5d-d884-366b-5897812afbd0@gmx.com>
 <mailman.80.1479960005.3146.python-list@python.org>
 <e4c9f90e-3ef0-41b1-b729-a48792ebada9@googlegroups.com>
 <CAPjBxg9LrwO3t2VVEs9p+fTL8a6HuCLbyF3rbMAqOU+e=0wNGQ@mail.gmail.com>
 <mailman.82.1479964961.3146.python-list@python.org>
 <a6a76fae-2371-40f7-abed-186a6f1fc023@googlegroups.com>
Message-ID: <gmDZz.29368$9u5.8341@fx43.am4>

On Thu, 24 Nov 2016 06:00:20 -0800, Cai Gengyang wrote:

> CaiGengYangs-MacBook-Pro:~ CaiGengYang$ import pygame -bash: import:
> command not found
> 
> 
> 
please do not top post as it makes the threads difficult to follow
the preferred style is interleave posing
(posting a reply after the text you are replying to)
and optionally snipping irrelevant parts of the post


-- 
"I'm growing older, but not up."
-- Jimmy Buffett


From gengyangcai at gmail.com  Thu Nov 24 10:09:22 2016
From: gengyangcai at gmail.com (Cai Gengyang)
Date: Thu, 24 Nov 2016 07:09:22 -0800 (PST)
Subject: NameError
In-Reply-To: <gmDZz.29368$9u5.8341@fx43.am4>
References: <223a00e3-1e8b-4ba3-9402-37a7f8ba9dcf@googlegroups.com>
 <CAPM-O+ySA-o3Afh12NU6Ce0REC_pnoH7_we4AOE-NYe9pSqOTw@mail.gmail.com>
 <mailman.77.1479956046.3146.python-list@python.org>
 <079079bb-6abd-4660-be25-dd4f33165e35@googlegroups.com>
 <33bcdbba-9c5d-d884-366b-5897812afbd0@gmx.com>
 <mailman.80.1479960005.3146.python-list@python.org>
 <e4c9f90e-3ef0-41b1-b729-a48792ebada9@googlegroups.com>
 <CAPjBxg9LrwO3t2VVEs9p+fTL8a6HuCLbyF3rbMAqOU+e=0wNGQ@mail.gmail.com>
 <mailman.82.1479964961.3146.python-list@python.org>
 <a6a76fae-2371-40f7-abed-186a6f1fc023@googlegroups.com>
 <gmDZz.29368$9u5.8341@fx43.am4>
Message-ID: <777cb0ce-b520-49c2-83b6-d4024fe8d05f@googlegroups.com>

Hmm, so whats the solution ?



On Thursday, November 24, 2016 at 11:07:05 PM UTC+8, alister wrote:
> On Thu, 24 Nov 2016 06:00:20 -0800, Cai Gengyang wrote:
> 
> > CaiGengYangs-MacBook-Pro:~ CaiGengYang$ import pygame -bash: import:
> > command not found
> > 
> > 
> > 
> please do not top post as it makes the threads difficult to follow
> the preferred style is interleave posing
> (posting a reply after the text you are replying to)
> and optionally snipping irrelevant parts of the post
> 
> 
> -- 
> "I'm growing older, but not up."
> -- Jimmy Buffett



From alister.ware at ntlworld.com  Thu Nov 24 10:21:57 2016
From: alister.ware at ntlworld.com (alister)
Date: Thu, 24 Nov 2016 15:21:57 GMT
Subject: NameError
References: <223a00e3-1e8b-4ba3-9402-37a7f8ba9dcf@googlegroups.com>
 <CAPM-O+ySA-o3Afh12NU6Ce0REC_pnoH7_we4AOE-NYe9pSqOTw@mail.gmail.com>
 <mailman.77.1479956046.3146.python-list@python.org>
 <079079bb-6abd-4660-be25-dd4f33165e35@googlegroups.com>
 <33bcdbba-9c5d-d884-366b-5897812afbd0@gmx.com>
 <mailman.80.1479960005.3146.python-list@python.org>
 <e4c9f90e-3ef0-41b1-b729-a48792ebada9@googlegroups.com>
 <CAPjBxg9LrwO3t2VVEs9p+fTL8a6HuCLbyF3rbMAqOU+e=0wNGQ@mail.gmail.com>
 <mailman.82.1479964961.3146.python-list@python.org>
 <a6a76fae-2371-40f7-abed-186a6f1fc023@googlegroups.com>
 <gmDZz.29368$9u5.8341@fx43.am4>
 <777cb0ce-b520-49c2-83b6-d4024fe8d05f@googlegroups.com>
Message-ID: <pADZz.29369$9u5.25586@fx43.am4>

On Thu, 24 Nov 2016 07:09:22 -0800, Cai Gengyang wrote:

> Hmm, so whats the solution ?
> 
>
The solution is to bottom post or interleave post as previously stated
 
> 
> On Thursday, November 24, 2016 at 11:07:05 PM UTC+8, alister wrote:
>> On Thu, 24 Nov 2016 06:00:20 -0800, Cai Gengyang wrote:
>> 
>> > CaiGengYangs-MacBook-Pro:~ CaiGengYang$ import pygame -bash: import:
>> > command not found
>> > 
>> > 
>> > 
>> please do not top post as it makes the threads difficult to follow the
>> preferred style is interleave posing (posting a reply after the text
>> you are replying to)
>> and optionally snipping irrelevant parts of the post
>> 
>> 
>> --
>> "I'm growing older, but not up."
>> -- Jimmy Buffett





-- 
There was a little girl
Who had a little curl
Right in the middle of her forehead.
When she was good, she was very, very good
And when she was bad, she was very, very popular.
		-- Max Miller, "The Max Miller Blue Book"


From gengyangcai at gmail.com  Thu Nov 24 10:25:37 2016
From: gengyangcai at gmail.com (Cai Gengyang)
Date: Thu, 24 Nov 2016 07:25:37 -0800 (PST)
Subject: NameError
In-Reply-To: <pADZz.29369$9u5.25586@fx43.am4>
References: <223a00e3-1e8b-4ba3-9402-37a7f8ba9dcf@googlegroups.com>
 <CAPM-O+ySA-o3Afh12NU6Ce0REC_pnoH7_we4AOE-NYe9pSqOTw@mail.gmail.com>
 <mailman.77.1479956046.3146.python-list@python.org>
 <079079bb-6abd-4660-be25-dd4f33165e35@googlegroups.com>
 <33bcdbba-9c5d-d884-366b-5897812afbd0@gmx.com>
 <mailman.80.1479960005.3146.python-list@python.org>
 <e4c9f90e-3ef0-41b1-b729-a48792ebada9@googlegroups.com>
 <CAPjBxg9LrwO3t2VVEs9p+fTL8a6HuCLbyF3rbMAqOU+e=0wNGQ@mail.gmail.com>
 <mailman.82.1479964961.3146.python-list@python.org>
 <a6a76fae-2371-40f7-abed-186a6f1fc023@googlegroups.com>
 <gmDZz.29368$9u5.8341@fx43.am4>
 <777cb0ce-b520-49c2-83b6-d4024fe8d05f@googlegroups.com>
 <pADZz.29369$9u5.25586@fx43.am4>
Message-ID: <a83b88bf-b5bc-4ff0-80bb-14e20b8a2dcb@googlegroups.com>

I mean whats the solution to import pygame ? How to import pygame ?


On Thursday, November 24, 2016 at 11:22:07 PM UTC+8, alister wrote:
> On Thu, 24 Nov 2016 07:09:22 -0800, Cai Gengyang wrote:
> 
> > Hmm, so whats the solution ?
> > 
> >
> The solution is to bottom post or interleave post as previously stated
>  
> > 
> > On Thursday, November 24, 2016 at 11:07:05 PM UTC+8, alister wrote:
> >> On Thu, 24 Nov 2016 06:00:20 -0800, Cai Gengyang wrote:
> >> 
> >> > CaiGengYangs-MacBook-Pro:~ CaiGengYang$ import pygame -bash: import:
> >> > command not found
> >> > 
> >> > 
> >> > 
> >> please do not top post as it makes the threads difficult to follow the
> >> preferred style is interleave posing (posting a reply after the text
> >> you are replying to)
> >> and optionally snipping irrelevant parts of the post
> >> 
> >> 
> >> --
> >> "I'm growing older, but not up."
> >> -- Jimmy Buffett
> 
> 
> 
> 
> 
> -- 
> There was a little girl
> Who had a little curl
> Right in the middle of her forehead.
> When she was good, she was very, very good
> And when she was bad, she was very, very popular.
> 		-- Max Miller, "The Max Miller Blue Book"


From gengyangcai at gmail.com  Thu Nov 24 10:30:02 2016
From: gengyangcai at gmail.com (Cai Gengyang)
Date: Thu, 24 Nov 2016 07:30:02 -0800 (PST)
Subject: NameError
In-Reply-To: <o170p0$pp3$2@dont-email.me>
References: <223a00e3-1e8b-4ba3-9402-37a7f8ba9dcf@googlegroups.com>
 <CAPM-O+ySA-o3Afh12NU6Ce0REC_pnoH7_we4AOE-NYe9pSqOTw@mail.gmail.com>
 <mailman.77.1479956046.3146.python-list@python.org>
 <079079bb-6abd-4660-be25-dd4f33165e35@googlegroups.com>
 <33bcdbba-9c5d-d884-366b-5897812afbd0@gmx.com>
 <mailman.80.1479960005.3146.python-list@python.org>
 <e4c9f90e-3ef0-41b1-b729-a48792ebada9@googlegroups.com>
 <CAPjBxg9LrwO3t2VVEs9p+fTL8a6HuCLbyF3rbMAqOU+e=0wNGQ@mail.gmail.com>
 <mailman.82.1479964961.3146.python-list@python.org>
 <a6a76fae-2371-40f7-abed-186a6f1fc023@googlegroups.com>
 <gmDZz.29368$9u5.8341@fx43.am4>
 <777cb0ce-b520-49c2-83b6-d4024fe8d05f@googlegroups.com>
 <o170p0$pp3$2@dont-email.me>
Message-ID: <5520a648-fbdc-432f-a10f-4f615218c223@googlegroups.com>

CaiGengYangs-MacBook-Pro:~ CaiGengYang$ pip install pygame
Collecting pygame
  Could not find a version that satisfies the requirement pygame (from versions: )
No matching distribution found for pygame
You are using pip version 7.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
CaiGengYangs-MacBook-Pro:~ CaiGengYang$ python
Python 2.7.10 (v2.7.10:15c95b7d81dc, May 23 2015, 09:33:12) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.




On Thursday, November 24, 2016 at 11:26:49 PM UTC+8, DFS wrote:
> On 11/24/2016 10:09 AM, Cai Gengyang wrote:
> 
> > Hmm, so whats the solution ?
> 
> 
> One solution - not the solution - is:
> 
> $ pip install pygame
> Collecting pygame
>    Downloading pygame-1.9.2b1-cp27-cp27m-win32.whl (4.3MB)
>      100% |################################| 4.3MB 181kB/s
> Installing collected packages: pygame
> Successfully installed pygame-1.9.2b1
> 
> $ python
> Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:19:22) [MSC v.1500 
> 32 bit (Intel)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
> 
> >>> import pygame
> >>>


From twgrops at googlemail.com  Thu Nov 24 10:45:30 2016
From: twgrops at googlemail.com (Thomas Grops)
Date: Thu, 24 Nov 2016 07:45:30 -0800 (PST)
Subject: Help with two issues, buttons and second class object
In-Reply-To: <mailman.90.1479999218.3146.python-list@python.org>
References: <f44a5197-0c3d-4a19-aad9-983244c8aa26@googlegroups.com>
 <o16ut3$78m$1@blaine.gmane.org>
 <mailman.90.1479999218.3146.python-list@python.org>
Message-ID: <dd7816bb-bbe8-4e27-b66d-3275d01a35f6@googlegroups.com>

Wow thankyou that code is really good, I had no programming knowledge until 2 months ago, I enjoy your descriptions it is really helpful for me. I like to understand what the code does before using it myself or a variant of it.

Will tweak bits tonight the project is in tomorrow. This code is just a side to the one I completed and will be handing in so im just doing this for fun and learning but the past months but as I learn new things I keep wanting to rebuild my program. I will be adding obstacles into the code too and eventually enemies, that explains the commented out code for attack and medic.

I will upload the code when I am done with it or get stuck again to see what your feedback is :D


From robert.snoeberger at gmail.com  Thu Nov 24 11:32:33 2016
From: robert.snoeberger at gmail.com (Robert Snoeberger)
Date: Thu, 24 Nov 2016 08:32:33 -0800 (PST)
Subject: Options for stdin and stdout when using pdb debugger
Message-ID: <baaf02f9-7c6c-4450-9e87-77ddacb5dcaa@googlegroups.com>

I would like to use pdb in an application where it isn't possible to use sys.stdin for input. I've read in the documentation for pdb.Pdb that a file object can be used instead of sys.stdin. Unfortunately, I'm not clear about my options for the file object. 

I've looked at rpdb on PyPI, which re-routes stdin and stdout to a socket handler. I can connect to the socket with telnet. This works well. 

I'm just curious if there are other options?

Thank you!


From alister.ware at ntlworld.com  Thu Nov 24 11:59:53 2016
From: alister.ware at ntlworld.com (alister)
Date: Thu, 24 Nov 2016 16:59:53 GMT
Subject: NameError
References: <223a00e3-1e8b-4ba3-9402-37a7f8ba9dcf@googlegroups.com>
 <CAPM-O+ySA-o3Afh12NU6Ce0REC_pnoH7_we4AOE-NYe9pSqOTw@mail.gmail.com>
 <mailman.77.1479956046.3146.python-list@python.org>
 <079079bb-6abd-4660-be25-dd4f33165e35@googlegroups.com>
 <33bcdbba-9c5d-d884-366b-5897812afbd0@gmx.com>
 <mailman.80.1479960005.3146.python-list@python.org>
 <e4c9f90e-3ef0-41b1-b729-a48792ebada9@googlegroups.com>
 <CAPjBxg9LrwO3t2VVEs9p+fTL8a6HuCLbyF3rbMAqOU+e=0wNGQ@mail.gmail.com>
 <mailman.82.1479964961.3146.python-list@python.org>
 <a6a76fae-2371-40f7-abed-186a6f1fc023@googlegroups.com>
 <gmDZz.29368$9u5.8341@fx43.am4>
 <777cb0ce-b520-49c2-83b6-d4024fe8d05f@googlegroups.com>
 <pADZz.29369$9u5.25586@fx43.am4>
 <a83b88bf-b5bc-4ff0-80bb-14e20b8a2dcb@googlegroups.com>
Message-ID: <d0FZz.15638$eh.6162@fx08.am4>

On Thu, 24 Nov 2016 07:25:37 -0800, Cai Gengyang wrote:

> I mean whats the solution to import pygame ? How to import pygame ?
> 
> 
> On Thursday, November 24, 2016 at 11:22:07 PM UTC+8, alister wrote:
>> On Thu, 24 Nov 2016 07:09:22 -0800, Cai Gengyang wrote:
>> 
>> > Hmm, so whats the solution ?
>> > 
>> > 
>> The solution is to bottom post or interleave post as previously stated
>>  
>>  
>> > On Thursday, November 24, 2016 at 11:07:05 PM UTC+8, alister wrote:
>> >> On Thu, 24 Nov 2016 06:00:20 -0800, Cai Gengyang wrote:
>> >> 
>> >> > CaiGengYangs-MacBook-Pro:~ CaiGengYang$ import pygame -bash:
>> >> > import:
>> >> > command not found
>> >> > 
>> >> > 
>> >> > 
>> >> please do not top post as it makes the threads difficult to follow
>> >> the preferred style is interleave posing (posting a reply after the
>> >> text you are replying to)
>> >> and optionally snipping irrelevant parts of the post
>> >> 
>> >> 
>> >> --
>> >> "I'm growing older, but not up."
>> >> -- Jimmy Buffett
>> 
>> 
>> 
>> 
>> 
>> --
>> There was a little girl Who had a little curl Right in the middle of
>> her forehead.
>> When she was good, she was very, very good And when she was bad, she
>> was very, very popular.
>> 		-- Max Miller, "The Max Miller Blue Book"



if you would not persist in ignoring a polite request to follow the 
posting conventions of this group I might be inclined to tel you where 
you are going wrong
-- 
Every country has the government it deserves.
		-- Joseph De Maistre


From torriem at gmail.com  Thu Nov 24 12:45:40 2016
From: torriem at gmail.com (Michael Torrie)
Date: Thu, 24 Nov 2016 10:45:40 -0700
Subject: NameError
In-Reply-To: <600d3d71-4b38-4196-b5b2-6c96aa7cfbce@googlegroups.com>
References: <223a00e3-1e8b-4ba3-9402-37a7f8ba9dcf@googlegroups.com>
 <CAPM-O+ySA-o3Afh12NU6Ce0REC_pnoH7_we4AOE-NYe9pSqOTw@mail.gmail.com>
 <mailman.77.1479956046.3146.python-list@python.org>
 <079079bb-6abd-4660-be25-dd4f33165e35@googlegroups.com>
 <33bcdbba-9c5d-d884-366b-5897812afbd0@gmx.com>
 <mailman.80.1479960005.3146.python-list@python.org>
 <e4c9f90e-3ef0-41b1-b729-a48792ebada9@googlegroups.com>
 <CAPjBxg9LrwO3t2VVEs9p+fTL8a6HuCLbyF3rbMAqOU+e=0wNGQ@mail.gmail.com>
 <mailman.82.1479964961.3146.python-list@python.org>
 <a6a76fae-2371-40f7-abed-186a6f1fc023@googlegroups.com>
 <b30df68c-7e71-2a78-21aa-4bf3e505d8bb@gmx.com>
 <mailman.91.1479999510.3146.python-list@python.org>
 <600d3d71-4b38-4196-b5b2-6c96aa7cfbce@googlegroups.com>
Message-ID: <9ef95af4-5f1b-eb70-c828-2668c095be56@gmail.com>

As alister said, please do not just hit reply and type your message at
the top.  Instead, place your reply below the quoted text you are
replying too.  This is not hard.  I realize there's a language barrier,
but please patiently read what alister said and understand what he's
saying. I know you're impatient to get Python working, but take a few
minutes to understand what we're saying.  See below for my reply to your
message (and an example of posting below quoted text).

On 11/24/2016 08:05 AM, Cai Gengyang wrote:
> CaiGengYangs-MacBook-Pro:~ CaiGengYang$ python
> Python 2.7.10 (v2.7.10:15c95b7d81dc, May 23 2015, 09:33:12) 
> [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import pygame
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> ImportError: No module named pygame
>>>> "import pygame"
> 'import pygame'
>>>>

Quite likely you are not running the version of Python that was
installed with brew.  Instead you are running the system version of
python that came with OS X, which is likely at /usr/bin/python.  The
Brew-installed python is, if I recall correctly, installed to
/usr/local/bin, and is Python 3.  So the correct command-line input
would be:

/usr/local/bin/python3

That should bring up the python prompt and if you were successful with
brew installing pygame, you should be able to import pygame and not get
an ImportError.



From gengyangcai at gmail.com  Thu Nov 24 13:45:40 2016
From: gengyangcai at gmail.com (Cai Gengyang)
Date: Thu, 24 Nov 2016 10:45:40 -0800 (PST)
Subject: NameError
In-Reply-To: <mailman.0.1480009545.2073.python-list@python.org>
References: <223a00e3-1e8b-4ba3-9402-37a7f8ba9dcf@googlegroups.com>
 <CAPM-O+ySA-o3Afh12NU6Ce0REC_pnoH7_we4AOE-NYe9pSqOTw@mail.gmail.com>
 <mailman.77.1479956046.3146.python-list@python.org>
 <079079bb-6abd-4660-be25-dd4f33165e35@googlegroups.com>
 <33bcdbba-9c5d-d884-366b-5897812afbd0@gmx.com>
 <mailman.80.1479960005.3146.python-list@python.org>
 <e4c9f90e-3ef0-41b1-b729-a48792ebada9@googlegroups.com>
 <CAPjBxg9LrwO3t2VVEs9p+fTL8a6HuCLbyF3rbMAqOU+e=0wNGQ@mail.gmail.com>
 <mailman.82.1479964961.3146.python-list@python.org>
 <a6a76fae-2371-40f7-abed-186a6f1fc023@googlegroups.com>
 <b30df68c-7e71-2a78-21aa-4bf3e505d8bb@gmx.com>
 <mailman.91.1479999510.3146.python-list@python.org>
 <600d3d71-4b38-4196-b5b2-6c96aa7cfbce@googlegroups.com>
 <9ef95af4-5f1b-eb70-c828-2668c095be56@gmail.com>
 <mailman.0.1480009545.2073.python-list@python.org>
Message-ID: <3420e3ff-958f-4939-9a7e-1adec13e3de8@googlegroups.com>

This is what I got :

CaiGengYangs-MacBook-Pro:~ CaiGengYang$ /usr/local/bin/python3 
Python 3.5.1 (v3.5.1:37a07cee5969, Dec  5 2015, 21:12:44) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pygame
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named 'pygame'
>>> 








On Friday, November 25, 2016 at 1:46:10 AM UTC+8, Michael Torrie wrote:
> As alister said, please do not just hit reply and type your message at
> the top.  Instead, place your reply below the quoted text you are
> replying too.  This is not hard.  I realize there's a language barrier,
> but please patiently read what alister said and understand what he's
> saying. I know you're impatient to get Python working, but take a few
> minutes to understand what we're saying.  See below for my reply to your
> message (and an example of posting below quoted text).
> 
> On 11/24/2016 08:05 AM, Cai Gengyang wrote:
> > CaiGengYangs-MacBook-Pro:~ CaiGengYang$ python
> > Python 2.7.10 (v2.7.10:15c95b7d81dc, May 23 2015, 09:33:12) 
> > [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
> > Type "help", "copyright", "credits" or "license" for more information.
> >>>> import pygame
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in <module>
> > ImportError: No module named pygame
> >>>> "import pygame"
> > 'import pygame'
> >>>>
> 
> Quite likely you are not running the version of Python that was
> installed with brew.  Instead you are running the system version of
> python that came with OS X, which is likely at /usr/bin/python.  The
> Brew-installed python is, if I recall correctly, installed to
> /usr/local/bin, and is Python 3.  So the correct command-line input
> would be:
> 
> /usr/local/bin/python3
> 
> That should bring up the python prompt and if you were successful with
> brew installing pygame, you should be able to import pygame and not get
> an ImportError.


From wants at no.mail.local  Thu Nov 24 14:46:58 2016
From: wants at no.mail.local (andy)
Date: Thu, 24 Nov 2016 20:46:58 +0100
Subject: printing funny symbols within spyder ide
Message-ID: <e872f$583743b2$54817cc5$2721@news1.surfino.com>

when printing these 'escaped-number-texts' within Linux/Spyder3 ide 
console, i get funny symbols like a "phone-symbol", triangles or symbols 
for male or female.

>>> print("\7") # gives a phone
>>> print("\5")
>>> print("\1")
>>> print("\21")
>>> print("\30")
>>> print("\31")
>>> print("\32")

def phonesymbol():
	print("\7")

phonesymbol() # prints a phone symbol

my question is, is this a bug or a feature of Spyder3?
if it is a feature, where can I find the complete table of symbols?

within bash - starting python3 - i get some ugly "unicode?" symbols, 
which look like a square containig tiny alphanumeric numbers.

best regards
andy


From spluque at gmail.com  Thu Nov 24 15:21:21 2016
From: spluque at gmail.com (Seb)
Date: Thu, 24 Nov 2016 14:21:21 -0600
Subject: generating list of files matching condition
References: <87fumh4jhj.fsf@otaria.sebmel.org> <o16b8q$a84$1@blaine.gmane.org>
Message-ID: <87vavcaata.fsf@gmail.com>

On Thu, 24 Nov 2016 10:18:21 +0100,
Peter Otten <__peter__ at web.de> wrote:

> Is win_files huge? Then it might help to avoid going over the entire
> list for every interval. Instead you can sort the list and then add to
> the current list while you are below nextw.

> My pandas doesn't seem to have Timedelta (probably it's too old), so
> here's a generic solution using only the stdlib:

[...]

> PS: If the files' prefixes differ you cannot sort by name. Instead use

> ec_files.sort(key=filename_to_time)

> PPS: There is probably a way to do this by converting the list to a
> pandas dataframe; it might be worthwhile to ask in a specialised
> forum.

Thank you, I learned a few things there (particularly nonlocal, since
I'm still in Python 2.7)!

-- 
Seb



From orgnut at yahoo.com  Thu Nov 24 15:37:11 2016
From: orgnut at yahoo.com (Larry Hudson)
Date: Thu, 24 Nov 2016 12:37:11 -0800
Subject: Help with two issues, buttons and second class object
In-Reply-To: <mailman.90.1479999218.3146.python-list@python.org>
References: <f44a5197-0c3d-4a19-aad9-983244c8aa26@googlegroups.com>
 <o16ut3$78m$1@blaine.gmane.org>
 <mailman.90.1479999218.3146.python-list@python.org>
Message-ID: <7vidnRXIhZzl0qrFnZ2dnUU7-UHNnZ2d@giganews.com>

On 11/24/2016 06:53 AM, Peter Otten wrote:
> Thomas Grops via Python-list wrote:
[snip...]
> Instead of repeating your code with copy-and-past make a helper function
> like the randx() posted by Larry Hudson.
>
> By the way, randint(min, max) may return max so there are 9 possible
> outcomes while you handle only 8. To be consistent with Python's half-open
> ranges use
>
> random.randrange(8) # may return 0, 1, 2, 3, 4, 5, 6, 7, but not 8
>

About my (Larry Hudson) randx() example code:

I used randint() in it because that was what the OP was using.  I didn't want to bother with the 
side issue of randint() vs randrange() since it was just meant as example code to be adapted. 
If I had written it for my own use I would have used randrange() for the consistency that you 
are describing.

Not an excuse ? just a comment.   ;-)

-- 
      -=- Larry -=-


From dieter at handshake.de  Fri Nov 25 02:21:40 2016
From: dieter at handshake.de (dieter)
Date: Fri, 25 Nov 2016 08:21:40 +0100
Subject: Options for stdin and stdout when using pdb debugger
References: <baaf02f9-7c6c-4450-9e87-77ddacb5dcaa@googlegroups.com>
Message-ID: <878ts83tyz.fsf@handshake.de>

Robert Snoeberger <robert.snoeberger at gmail.com> writes:

> I would like to use pdb in an application where it isn't possible to use sys.stdin for input. I've read in the documentation for pdb.Pdb that a file object can be used instead of sys.stdin. Unfortunately, I'm not clear about my options for the file object. 

Likely, it must be something with a "readline" method. This would mean
that the restrictions are very weak.



From list at qtrac.plus.com  Fri Nov 25 03:29:38 2016
From: list at qtrac.plus.com (Mark Summerfield)
Date: Fri, 25 Nov 2016 00:29:38 -0800 (PST)
Subject: The Case Against Python 3
In-Reply-To: <75df94b8-8af4-4de4-93a8-dd4de5d186b7@googlegroups.com>
References: <75df94b8-8af4-4de4-93a8-dd4de5d186b7@googlegroups.com>
Message-ID: <b44e089f-c7e4-4fd7-8c78-585b8b02269a@googlegroups.com>

On Thursday, November 24, 2016 at 7:35:03 PM UTC, bream... at gmail.com wrote:
> It's all here https://learnpythonthehardway.org/book/nopython3.html although I strongly suggest that people have large piles of sedatives to hand before reading the article.  Does me a favour though, i've been looking for another member of my dream team for some time.
> 
> Kindest regards.
> 
> Mark Lawrence.

I think the article is full of factual errors and is at best misleading and at worst downright dishonest in its criticisms of Python 3. (My only serious complaint about Python is that it doesn't support a key-ordered dict.) Anyway, here are my initial responses to some of the articles "points".

The article claims:

    "There is a high probability that Python 3 is such a failure it will kill Python."

This is despite the fact that there are more Python programmers than ever before, that there is more investment in Python programming, tools, and libraries than ever before, and that more and more new projects start out as Python 3 only.

The article says:

    "Also, it's been over a decade, maybe even multiple decades, and Python 3 still isn't above about 30% in adoption."

Python 3.0 was released in 2008, so it hasn't been a single decade. According to an analysis of PyPI downloads, Python 2-only packages have steadily declined in the past 5 years with Python 3-only packages steadily rising and about to overtake, and with a steady rise in packages that support both:
https://blogs.msdn.microsoft.com/pythonengineering/2016/03/08/python-3-is-winning/
The Python 3 Readiness web site:
http://py3readiness.org/
shows that today, 341 out of 360 of Python's most popular packages are Python 3 compatible.

The article says:

    "The fact that you can't run Python 2 and Python 3 at the same time"

This is untrue. Most Linux distributions not only ship with Python 2 and Python 3, but often end up with both installed. Modern versions of Fedora and Ubuntu depend on Python 3, but allow Python 2 to be installed side-by-side to support older software. On Windows it is easy to install both Python 2 and Python 3 and the py.exe that comes with Python 3 will detect which Python version is required and use it, if it is installed.

The article says:

    "Every time you attempt to deal with characters in your programs you'll have to understand the difference between byte sequences and Unicode strings."

If you are doing low-level networking or other low-level programming dealing with raw bytes this is true. But for application programming you just put your strings in quotes and they work fine. And if you're in a non-English environment, you still just put your strings in quotes and they still work fine. But in Python 2, for non-English environments you need to add an encoding comment at the top of each file and you may have to use Unicode escapes.

Working with strings in Python 3 is easier than in Python 2 since you always know they are Unicode whereas in Python 2 they can be in any 8-bit encoding depending on the encoding comment, the environment, or where they came from.

And working with raw bytes is easier in Python 3 because of the bytes and bytearray objects which are optimized for use with bytes rather than Python 2's jack of all trades str type.

The article has a section called:

    "Core Libraries Not Updated"

All of Python's core libraries were updated for Python 3, and further updating is an ongoing process. In general, if you have a library that can work with bytes or strings, the library will accept either and return the same type. So, for example, if you use the regex library, re, and give it a bytes object, any matches will be bytes objects, but if you give it string, any matches will be strings.

The article has a section called:

    "Python 3 Is Not Turing Complete"

This would be laughable if it wasn't being said straight. He claims that "Currently you cannot run Python 2 inside the Python 3 virtual machine." and that this 'proves' that Python 3 isn't Turing complete.

It is true that you can't run Python 2 code in a Python 3 interpreter (unless you use six), but that says nothing about whether Python 3 is Turing complete or not. Turing completeness depends on whether a language supports certain computational features, such as branching and unbounded looping, and Python 3 (and Python 2) have all the necessary features.

What he doesn't seem to understand is that Python 3 (or any other Turing complete language), can be used to _write an interepreter_ for any other language. So, a Python 2 interpreter could be written in Python 3, but no one has done so because there's no point when it is far easier to port Python 2 to Python 3, especially since Python 2 support will cease in a few years time.

The article has a section called:

    "Purposefully Crippled 2to3 Translator"

This section is incredibly insulting to the Python developers. The reason that 2to3 is imperfect is because like so many other things, it is fairly easy to get an 80% solution, but incredibly difficult to go much further. In addition, with the creation of the six library, it became possible to write code that was both Python 2 and Python 3 compatible, so people who want both can have it in the same code base without the need for 2to3. And those who don't need Python 2, don't need 2to3 (except maybe once to help with an initial port) or six.

The article has a section called:

    "Statically Typed Strings"

The title is wrong of course because Python uses dynamic typing. But his chief complaint seems to be that you can't mix strings and bytes in Python 3. That's a deliberate design choice that several Python core developers have explained. Essentially they are saying that you can't concatenate a bunch of raw bytes with a string in the same way that you can't add a set to a list -- and this makes perfect sense because raw bytes could be just bytes, or they could be a representation of text in which case by specifying the encoding (i.e., converting them to a string) the concatenation can take place. And this is in keeping with Python's core philosphy of being explicit.

The article has a section called:

    "Too Many Formatting Options"

He's right! The % formatting was kept to help port old code, the new .format() which is far more versatile is a bit verbose, so finally they've settled on f-strings. So, you do need to know that all three exist (e.g., for maintaining code), but you can easily choose the style that you prefer and just use that.

Conclusion

I have taught Python to many people and written books on Python. I started using Python 3 from the first alpha and by the first beta I'd (manually) ported all my own tools and utilities to it. I also develop commercial software, all of which is written in Python 3 which I find far easier to use in part _because_ of the clear text vs. bytes distinction, plus a number of other smaller improvements (unambiguous except blocks) etc. And certainly, I think that Python 3 is much easier to learn than Python 2. The first thing I always had to do when teaching Python 2 was explain why print was different from everything else (i.e., because it was a statement, not a function, and so didn't need parentheses around its arguments), but for Python 3 print() is just like a function you create yourself.

I've been using Python 3 very happily for 8 years now and it seems to get better and better with each new release.


From rustompmody at gmail.com  Fri Nov 25 03:45:20 2016
From: rustompmody at gmail.com (Rustom Mody)
Date: Fri, 25 Nov 2016 00:45:20 -0800 (PST)
Subject: A Rebuttal For Python 3
In-Reply-To: <3aaa8c9b-6f25-4a40-8378-aa51a6da352e@googlegroups.com>
References: <3aaa8c9b-6f25-4a40-8378-aa51a6da352e@googlegroups.com>
Message-ID: <cbe0720c-d54a-4682-b3c5-5206f5cff3a1@googlegroups.com>

On Friday, November 25, 2016 at 1:27:18 AM UTC+5:30, bream... at gmail.com wrote:
> https://eev.ee/blog/2016/11/23/a-rebuttal-for-python-3/ is one of presumably many responses to the article I posted about under the subject "The Case Against Python 3".  Enjoy :)

I'd say we are living in astrological? times?

There was a recent election?
In a grand and famous nation
The expletives xchanged
Midst ppl deranged
Let someone else complete the ditty

Considering that the last lines of the article usually contain the summary of the article.  And they run as follows:

|Fuck off...
|
| This is because you don?t know what you?re talking about, and you should shut up.

I am just going to comply (unless someone explain why so much uncivility is such a great idea)

> 
> Kindest regards.

You mean that?

----------------------------
? Which in more colloquial language means "its in the air".  Or its everywhere:
http://www.nytimes.com/2016/11/14/opinion/the-incendiary-appeal-of-demagoguery-in-our-time.html?_r=0


From __peter__ at web.de  Fri Nov 25 03:47:50 2016
From: __peter__ at web.de (Peter Otten)
Date: Fri, 25 Nov 2016 09:47:50 +0100
Subject: generating list of files matching condition
References: <87fumh4jhj.fsf@otaria.sebmel.org> <o16b8q$a84$1@blaine.gmane.org>
 <87vavcaata.fsf@gmail.com>
Message-ID: <o18trj$a8h$1@blaine.gmane.org>

Seb wrote:

> On Thu, 24 Nov 2016 10:18:21 +0100,
> Peter Otten <__peter__ at web.de> wrote:
> 
>> Is win_files huge? Then it might help to avoid going over the entire
>> list for every interval. Instead you can sort the list and then add to
>> the current list while you are below nextw.
> 
>> My pandas doesn't seem to have Timedelta (probably it's too old), so
>> here's a generic solution using only the stdlib:
> 
> [...]
> 
>> PS: If the files' prefixes differ you cannot sort by name. Instead use
> 
>> ec_files.sort(key=filename_to_time)
> 
>> PPS: There is probably a way to do this by converting the list to a
>> pandas dataframe; it might be worthwhile to ask in a specialised
>> forum.
> 
> Thank you, I learned a few things there (particularly nonlocal, since
> I'm still in Python 2.7)!

Did you get the code to work in Python 2? One way is to tack the nonlocal 
value as an attribute onto the first mutable you can find:

def make_key(delta_t):
    def key(filename):
        if key.upper_bound is None:
            key.upper_bound = filename_to_time(filename) + delta_t
        else:
            t = filename_to_time(filename)
            while t >= key.upper_bound:
                key.upper_bound += delta_t
        return key.upper_bound
    key.upper_bound = None
    return key

And if it worked, did it actually improve performance?



From ankur at numeratelabs.com  Fri Nov 25 04:02:34 2016
From: ankur at numeratelabs.com (Ankur Gupta)
Date: Fri, 25 Nov 2016 01:02:34 -0800 (PST)
Subject: Simple Python Quiz To Mark 100th Issue of ImportPython
Message-ID: <b6cf5b46-406a-42c2-9d18-5ccdd6841d73@googlegroups.com>

Hey Python Programmers,

Import Python Newsletter completed 2 years and 100 issues. Have a simple fun python quiz http://importpython.com/newsletter/quiz/ to mark the milestone. 

Happy ThanksGiving day to you all.

Regards,
Ankur


From rosuav at gmail.com  Fri Nov 25 04:31:10 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 25 Nov 2016 20:31:10 +1100
Subject: The Case Against Python 3
In-Reply-To: <b44e089f-c7e4-4fd7-8c78-585b8b02269a@googlegroups.com>
References: <75df94b8-8af4-4de4-93a8-dd4de5d186b7@googlegroups.com>
 <b44e089f-c7e4-4fd7-8c78-585b8b02269a@googlegroups.com>
Message-ID: <CAPTjJmrQyp=Bs2MmMM9+5aUf54XmgydcMADnODJ65USUD0GwRA@mail.gmail.com>

On Fri, Nov 25, 2016 at 7:29 PM, Mark Summerfield <list at qtrac.plus.com> wrote:
> The article has a section called:
>
>     "Statically Typed Strings"
>
> The title is wrong of course because Python uses dynamic typing. But his chief complaint seems to be that you can't mix strings and bytes in Python 3. That's a deliberate design choice that several Python core developers have explained. Essentially they are saying that you can't concatenate a bunch of raw bytes with a string in the same way that you can't add a set to a list -- and this makes perfect sense because raw bytes could be just bytes, or they could be a representation of text in which case by specifying the encoding (i.e., converting them to a string) the concatenation can take place. And this is in keeping with Python's core philosphy of being explicit.
>

It's worse than that. Look at his comparison of Py3 and Py2. I've
shortened them some to highlight the part I'm looking at:

x = bytes("hello", 'utf-8')
y = "hello"
def addstring(a, b):
    return a + b

addstring(x, y)
# TypeError

==========

def addstring(a, b):
    return a + b
x = "hello"
y = bytes("hello")
addstring(x, y)
# 'hellohello'

==========

He clearly does not understand the difference between bytes and text,
as has been proven earlier, but this demonstrates that he doesn't even
understand the difference between Python's data types. The first
example is trying to add a bytestring to a Unicode string; the second
is actually adding two byte strings. He could have given a demo of how
Python 2 lets you join str and unicode, but it would have spoiled his
Py2 code by putting u'hellohello' into his output, and making Py3
actually look better. Can't have that.

Then he says:
> If they're going to require beginners to struggle with the difference between bytes and Unicode the least they could do is tell people what variables are bytes and what variables are strings.
>

The trouble is, by the time you're adding bytes and text, you're not
looking at variables any more. You're looking at objects. I don't
think he's properly understood Python's object model.

Here, have some FUD:
> Strings are also most frequently received from an external source, such as a network socket, file, or similar input. This means that Python 3's statically typed strings and lack of static type safety will cause Python 3 applications to crash more often and have more security problems when compared with Python 2.
>

What security problems? Any evidence of that?

On the face of it, without any actual specific examples, which of
these would you expect to be more security-problem-prone: mixing data
types, or throwing exceptions? In a web application, an exception can
be caught at a high level, logged, and handled by kicking a 500 back
to the client. In other applications, there may be an equivalent, or
you just terminate the server (client gets disconnected) and start up
again. At worst, this means that someone can exploit the whole "crash
and restart" thing as a way to DOS you. Here, let me walk you through
some different numeric types, and you tell me which ones are equal and
which aren't - and the security implications of that:

1) 1e2 == 100 ?
2) 1e2 == "100" ?
3) "1e2" == 100 ?
4) "1e2" == "100" ?

#1 makes perfect sense. Python says, yes, this is the case. (Not all
languages will; 1e2 is a floating-point literal, 100 is an integer,
and it's conceivable to keep them separate.)

#2 is acceptable to languages with "sloppy comparison" and "strict
comparison" operators, like ECMAScript/JavaScript. The number 100 is
(non-strictly) equal to the string "100".

#3 depends on whether sloppy comparisons are done by converting to
string or converting to number. ECMAScript treats them as equal, but
I'm just as happy with that being false (actually, probably slightly
happier).

#4 makes no sense to any sane programmer [1], which must be why PHP
chose to have that one be true.

Security implications of two different hexadecimal strings comparing
equal.... that can't have any bearing on passwords now, can it...

Is b"hello" == u"hello" ever a security consideration? If it is, my
money is on the exception being the *more* secure option.

Straight-up false:
> The point being that character encoding detection and negotiation is a solved problem.

Nope, nope it isn't. One of my hobbies is collecting movie subtitles
in various languages [2]. They generally come to me in eight-bit
encodings with no declaration. Using only internal evidence, chardet
has about a 66% hit rate at a pass mark of "readable enough that I can
figure out the language", and a much lower hit rate at "actually the
correct encoding". With better heuristics (maybe a set of rules
specifically aimed at reading subtitle files), that could probably get
as far as 100% readable and 75% correct, but it is *never* going to be
perfect, because *the input is ambiguous*.

If other languages appear to have gotten this right, it's probably
because they either enforce a single encoding (eg UTF-8), or just
ignore the whole problem, assuming that someone else will have to deal
with it.

Mark says:
> He's right! The % formatting was kept to help port old code, the new .format() which is far more versatile is a bit verbose, so finally they've settled on f-strings. So, you do need to know that all three exist (e.g., for maintaining code), but you can easily choose the style that you prefer and just use that.
>

Not strictly true. An f-string is a special construct that isn't as
flexible as the other formatting types. You can read a bracey or
percent-marked string from a file, then interpolate it with values at
run time. You can't do that with an f-string, short of messing around
with eval (which is not as pretty as just msg.format(...), plus you
have to trust your external file as if it were code). This has strong
implications for i18n/l10n, and some other situations as well. So the
other string formatting facilities aren't ever going to die, and you
really should learn one of them.

I do agree that you're welcome to teach just one of them, though, and
worry about the other if and when it ever comes up. Personally, I
quite like percent-formatting, because it's the same as can be used in
a lot of other languages (including shell scripting, via GNU printf),
but brace-formatting lets you reorder the parameters, so it has
flexibility that can be important for i18n. So my conclusion would
probably be: Use f-strings for the simple cases where you'd be using a
literal, and then have a glance at each of the other two, so you know
they're there when the time comes.

He concludes that Py3 is still unusable because he keeps trying to
port code and failing. That says, to me, that he needs to take a step
back and learn about the fundamental difference between text and
bytes, and that might mean learning a bit of a language like Russian
or Japanese, where text obviously can't be squeezed into ASCII or into
a typical US-English eight-bit character set. For my part, though, I
can attest that *not one* of my students has had a problem with Py3,
ever since the course switched over. And that includes three (so far)
who, after a month and a half of learning JavaScript, are given five
days to learn Python and do something useful with it. Five days. They
start on Monday, and by Friday close-of-business, they demonstrate
what they've learned (in a group where all the students have learned
something in a week - eg Angular.js, Socket.io, React Native, Ruby,
mobile app design, etc), and it goes into the portfolio. Now, if
Python 3 were impossible to learn, you would expect these people to
struggle. They don't. In fact, as I was mentoring one of them, I kept
telling him "scope it back, scope it back, you have only X days to
finish this" - but he charged ahead and did everything anyway. Python
is pretty easy to learn; all three of my flex week students had moved
beyond messing with the language before the end of Monday, and were
onto actual productive work on the project.

And it's probably even easier for a perfectly new programmer to
understand. You ignore bytes altogether until you start working with
networks (even with disks, you can read and write in text mode) or
actual binary data (graphic file formats or something). Text behaves
the way you'd expect text to. You can use English variable names - but
you can also use French, or Swedish, or Russian, or Japanese, because
Python doesn't restrict you to ASCII. It does exactly what you'd
expect. You just have to expect based on a human's outlook, rather
than a C programmer's.

ChrisA

[1] I am, however, open to the argument that computer programmers are
by definition not sane.
[2] eg for Disney's "Frozen":
https://github.com/Rosuav/LetItTrans/tree/master/entire


From twgrops at googlemail.com  Fri Nov 25 04:36:23 2016
From: twgrops at googlemail.com (Thomas Grops)
Date: Fri, 25 Nov 2016 01:36:23 -0800 (PST)
Subject: Help with two issues, buttons and second class object
In-Reply-To: <mailman.90.1479999218.3146.python-list@python.org>
References: <f44a5197-0c3d-4a19-aad9-983244c8aa26@googlegroups.com>
 <o16ut3$78m$1@blaine.gmane.org>
 <mailman.90.1479999218.3146.python-list@python.org>
Message-ID: <193607fb-a610-4eff-b17b-3baafd370a84@googlegroups.com>

Peter, in your code what does that self.root = root mean in the __init__ function of the class


From twgrops at googlemail.com  Fri Nov 25 04:40:41 2016
From: twgrops at googlemail.com (Thomas Grops)
Date: Fri, 25 Nov 2016 01:40:41 -0800 (PST)
Subject: Help with two issues, buttons and second class object
In-Reply-To: <f44a5197-0c3d-4a19-aad9-983244c8aa26@googlegroups.com>
References: <f44a5197-0c3d-4a19-aad9-983244c8aa26@googlegroups.com>
Message-ID: <c1c64ef2-c21a-4990-8856-9a62f79ea266@googlegroups.com>

Also I am struggling to understand:

   def move_tank(self, dx, dy): 
        self.x += dx 
        self.y += dy 
        self.canvas.move(self.id, dx, dy) 

Where does the dx and dy values get input?


From rosuav at gmail.com  Fri Nov 25 04:52:11 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 25 Nov 2016 20:52:11 +1100
Subject: Simple Python Quiz To Mark 100th Issue of ImportPython
In-Reply-To: <b6cf5b46-406a-42c2-9d18-5ccdd6841d73@googlegroups.com>
References: <b6cf5b46-406a-42c2-9d18-5ccdd6841d73@googlegroups.com>
Message-ID: <CAPTjJmpz68+EnYYW5psHb0Qcx_AKb7woqUJdi6z0cf8YfeJvmw@mail.gmail.com>

On Fri, Nov 25, 2016 at 8:02 PM, Ankur Gupta <ankur at numeratelabs.com> wrote:
> Import Python Newsletter completed 2 years and 100 issues. Have a simple fun python quiz http://importpython.com/newsletter/quiz/ to mark the milestone.

With your question about laying out code, are you aware that PEP 8
specifically allows both?

"""
While sometimes it's okay to put an if/for/while with a small body on
the same line, never do this for multi-clause statements. Also avoid
folding such long lines!
"""

You have a very simple condition (actually, trivially simple - in its
exact form as given, it's always true) guarding a simple statement.
There's nothing wrong with putting that on a single line. I especially
consider this important when there are multiple parallel conditions:

if foo.x: foo.widen()
if foo.y: foo.heighten()
if foo.z: foo.deepen()

There's no point breaking these into two lines each; they show a
perfect parallel as it is, and splitting the lines would disrupt that.

ChrisA


From barthelemy at geovariances.com  Fri Nov 25 05:52:09 2016
From: barthelemy at geovariances.com (Olivier Barthelemy)
Date: Fri, 25 Nov 2016 02:52:09 -0800 (PST)
Subject: Cohabitation of debug and release python in the same folder
Message-ID: <692938df-730d-4e4e-8adf-48b52e7179f9@googlegroups.com>

If i try to build a normal python and a 'with-pydebug' python in the same folder, I can run both versions of python by running the real executable names instead of the generic name symlinks.

However, if i want to use sysconfig.get_config_var() to check "Py_DEBUG" (to then know from .py files whether i need to load debug or release versions of other python modules that need c++ compilation), the variables seem to have the value of the last version i installed.
In my case, i compiled and installed in debug, then in release, so the default names symlinks would still point to a release version. And get_config_vars() called from the debug python binary seem to have the values of the release one.

Is there any way to work around that conflict, should i file a bug  for it, or is it "expected" and i should just install my two pythons in different folders?


From __peter__ at web.de  Fri Nov 25 05:58:27 2016
From: __peter__ at web.de (Peter Otten)
Date: Fri, 25 Nov 2016 11:58:27 +0100
Subject: Help with two issues, buttons and second class object
References: <f44a5197-0c3d-4a19-aad9-983244c8aa26@googlegroups.com>
 <o16ut3$78m$1@blaine.gmane.org>
 <mailman.90.1479999218.3146.python-list@python.org>
 <193607fb-a610-4eff-b17b-3baafd370a84@googlegroups.com>
Message-ID: <o195go$7uo$1@blaine.gmane.org>

Thomas Grops via Python-list wrote:

> Peter, in your code what does that self.root = root mean in the __init__
> function of the class

In your original script you used global variables to access the tkinter.Tk() 
object and the canvas. A simplified example:

import tkinter

class Tank:
    def make_button(self):
        tkinter.Button(root).pack()

root = tkinter.Tk() # you used the name "main" instead of "root"
tank = Tank()
tank.make_button()
root.mainloop()

Programmers with a little experience tend to avoid globals, so let's put the 
script code above into a function:

import tkinter

class Tank:
    def make_button(self):
        tkinter.Button(root).pack()

def main():
    root = tkinter.Tk() # you used the name "main" instead of "root"
    tank = Tank()
    tank.make_button()
    root.mainloop()

main()

When you run the above you'll get a NameError. Because make_button() 
accesses root which is no longer global you have to find a way to pass it to 
the tank instance. One way is to change the make_button() method and pass it 
along as an argument:

class Tank:
    def make_button(self, root):
        tkinter.Button(root).pack()

def main():
    root = tkinter.Tk() 
    tank = Tank()
    tank.make_button(root)
    root.mainloop()

The other way (the one I used) is to make it an attribute of the tank 
instance:

class Tank:
    def __init__(self, root):
        self.root = root
    def make_button(self):
        tkinter.Button(self.root).pack()

def main():
    root = tkinter.Tk() 
    tank = Tank(root)
    tank.make_button()
    root.mainloop()

Now you can access root as self.root in every method of the Tank class. 
There is no advantage in this case, but when root is referred in more than 
one place, either in the class or by the code invoking methods of the class, 
it will be more convenient.

An important advantage of using an explicit argument instead of a global is 
that different instances can have different values for root. For a Tank that 
might be battlefields

def main():
    root = tkinter.Tk()
    other = tkinter.Toplevel()

    for battlefield in root, other:
        tank = Tank(battlefield)
        tank.make_button()

    root.mainloop()

Note that this requires no change of the Tank class.



From fabien.maussion at gmail.com  Fri Nov 25 06:11:44 2016
From: fabien.maussion at gmail.com (Fabien)
Date: Fri, 25 Nov 2016 12:11:44 +0100
Subject: The Case Against Python 3
References: <75df94b8-8af4-4de4-93a8-dd4de5d186b7@googlegroups.com>
 <b44e089f-c7e4-4fd7-8c78-585b8b02269a@googlegroups.com>
Message-ID: <o1969f$1p1e$2@gioia.aioe.org>

On 11/25/2016 09:29 AM, Mark Summerfield wrote:
> On Thursday, November 24, 2016 at 7:35:03 PM UTC, bream... at gmail.com wrote:
>> > It's all here https://learnpythonthehardway.org/book/nopython3.html
> I think the article is full of factual errors and is at best misleading
 > and at worst downright dishonest in its criticisms of Python 3.

 > (snipped answers)

Thanks for your answers, which concur well with the post that Mark 
Lawrence linked earlier 
(https://eev.ee/blog/2016/11/23/a-rebuttal-for-python-3/ ).

I'd be interested to read what the community thinks about the fact that 
his book (learn the hard way) is extremely influential among beginners, 
and what tools do we have to avoid that beginners stumble across such 
opinions in their very first steps with python...



From __peter__ at web.de  Fri Nov 25 06:24:15 2016
From: __peter__ at web.de (Peter Otten)
Date: Fri, 25 Nov 2016 12:24:15 +0100
Subject: Help with two issues, buttons and second class object
References: <f44a5197-0c3d-4a19-aad9-983244c8aa26@googlegroups.com>
 <c1c64ef2-c21a-4990-8856-9a62f79ea266@googlegroups.com>
Message-ID: <o1970t$vg0$1@blaine.gmane.org>

Thomas Grops via Python-list wrote:

> Also I am struggling to understand:
> 
>    def move_tank(self, dx, dy):
>         self.x += dx
>         self.y += dy
>         self.canvas.move(self.id, dx, dy)
> 
> Where does the dx and dy values get input?

To find the place where the move_tank() method is invoked hit the search 
button or key of your text editor. In this case you'll find

    def move(self):
        self.move_tank(*self.moves[self.direction])
        ...

so move_tank() is invoked by the move() method. But what the heck is

*self.moves[self.direction]

? From the fact that the script runs without error you can conclude that it 
resolves to two integer values. Let's try in the interactive interpreter
(the script is called tanks2, so that's what I import):

$ python3
Python 3.4.3 (default, Nov 17 2016, 01:08:31) 
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tkinter
>>> import tanks2
>>> root = tkinter.Tk()
>>> canvas = tkinter.Canvas(root, width=100, height=100)
>>> canvas.pack()
>>> tank = tanks2.Tank(root, canvas, "example", 10, 10, 10, 10, "red")
>>> tank.direction
0

That's the value determining the direction into which the tank is supposed 
to move (what you called "count" in your script).

>>> tank.moves
[(5, 0), (-5, 0), (0, -2), (0, 2), (-1, -1), (1, -1), (1, 1), (-1, 1)]

That's the list of speed vectors I set up in the initialiser (the 
Tank.__init__() method)

>>> tank.moves[tank.direction]
(5, 0)

So move_tank() is supposed to move the tank 5 pixels to the right and 0 
pixels down. For this invocation

self.move_tank(*self.moves[self.direction])

is equivalent to

self.move_tank(*(5, 0))

The leading star tells python to treat the elements of the tuple as if they 
were passed individually to the function or method. The actual dx and dy are 
then 5 and 0:

self.move_tank(5, 0)

Now let's move our little red tank:

>>> tank.x, tank.y
(10, 10)
>>> tank.move_tank(3, 7)
>>> tank.x, tank.y
(13, 17)
>>> tank.move_tank(tank.moves[tanks2.MRIGHTDOWN])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: move_tank() missing 1 required positional argument: 'dy'

Oops, I forgot the leading *. Second attempt:

>>> tank.move_tank(*tank.moves[tanks2.MRIGHTDOWN])
>>> tank.x, tank.y
(14, 18)





From nikunjbadjatya at gmail.com  Fri Nov 25 06:24:37 2016
From: nikunjbadjatya at gmail.com (Nikunj)
Date: Fri, 25 Nov 2016 03:24:37 -0800 (PST)
Subject: Immutability of Floats, Ints and Strings in Python
Message-ID: <31b70ea6-d3ed-4756-81de-e6f327ebb56f@googlegroups.com>

Hi All,

Out of curiosity, I wanted to understand the reason behind having different memory location for two identical floats . This is unlike ints or strings. Tried googling but couldn't find anything concrete. Any links or references would be appreciated!

Example:

For FLOATS:
==========

>>> l = 1.3
>>> id(l)
140421602788216

>>> k = 1.3
>>> id(k)
140421602788240

>>> k == l
True

>>> k is l
False

For INTS and STRINGS:
=================
>>> i = 2
>>> o = 2
>>> id(i), id(o)
(140421602779712, 140421602779712)

>>> i is o
True

>>> a1 = 'hi'
>>> a2 = 'hi'
>>> a1 is a2
True



Thanks.


From ned at nedbatchelder.com  Fri Nov 25 06:33:32 2016
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Fri, 25 Nov 2016 03:33:32 -0800 (PST)
Subject: Immutability of Floats, Ints and Strings in Python
In-Reply-To: <31b70ea6-d3ed-4756-81de-e6f327ebb56f@googlegroups.com>
References: <31b70ea6-d3ed-4756-81de-e6f327ebb56f@googlegroups.com>
Message-ID: <4630ad9e-e336-4f78-aebb-706d403737ee@googlegroups.com>

On Friday, November 25, 2016 at 6:24:47 AM UTC-5, Nikunj wrote:
> Hi All,
> 
> Out of curiosity, I wanted to understand the reason behind having different memory location for two identical floats . This is unlike ints or strings. Tried googling but couldn't find anything concrete. Any links or references would be appreciated!
> 
> Example:
> 
> For FLOATS:
> ==========
> 
> >>> l = 1.3
> >>> id(l)
> 140421602788216
> 
> >>> k = 1.3
> >>> id(k)
> 140421602788240
> 
> >>> k == l
> True
> 
> >>> k is l
> False
> 
> For INTS and STRINGS:
> =================
> >>> i = 2
> >>> o = 2
> >>> id(i), id(o)
> (140421602779712, 140421602779712)
> 
> >>> i is o
> True
> 
> >>> a1 = 'hi'
> >>> a2 = 'hi'
> >>> a1 is a2
> True

It's more complicated than that:

    (using Python 2.7.10)
    >>> i = 2002
    >>> o = 2002
    >>> i is o
    False
    >>> a = 'what!?'
    >>> b = 'what!?'
    >>> a is b
    False

A Python implementation can choose when to reuse immutable objects and
when not to.  Reusing a value has a cost, because the values have to
be kept, and then found again. So the cost is only paid when there's
a reasonable chance that the values will actually be needed again.
And that cost has to be weighed against the opposite cost of simply
making a new object instead.

To answer your direct question: floats are so numerous and varied, it's
reasonable to guess that you won't often need the same one.  And the cost
to make a new one is so low, that it's simpler just to not reuse any of
them.

--Ned.


From ned at nedbatchelder.com  Fri Nov 25 06:37:32 2016
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Fri, 25 Nov 2016 03:37:32 -0800 (PST)
Subject: Immutability of Floats, Ints and Strings in Python
In-Reply-To: <4630ad9e-e336-4f78-aebb-706d403737ee@googlegroups.com>
References: <31b70ea6-d3ed-4756-81de-e6f327ebb56f@googlegroups.com>
 <4630ad9e-e336-4f78-aebb-706d403737ee@googlegroups.com>
Message-ID: <6d1b57ad-736d-4dc9-8dc3-d835fdb15ebc@googlegroups.com>

On Friday, November 25, 2016 at 6:34:00 AM UTC-5, Ned Batchelder wrote:
> On Friday, November 25, 2016 at 6:24:47 AM UTC-5, Nikunj wrote:
> > Hi All,
> > 
> > Out of curiosity, I wanted to understand the reason behind having different memory location for two identical floats . This is unlike ints or strings. Tried googling but couldn't find anything concrete. Any links or references would be appreciated!
> > 
> > Example:
> > 
> > For FLOATS:
> > ==========
> > 
> > >>> l = 1.3
> > >>> id(l)
> > 140421602788216
> > 
> > >>> k = 1.3
> > >>> id(k)
> > 140421602788240
> > 
> > >>> k == l
> > True
> > 
> > >>> k is l
> > False
> > 
> > For INTS and STRINGS:
> > =================
> > >>> i = 2
> > >>> o = 2
> > >>> id(i), id(o)
> > (140421602779712, 140421602779712)
> > 
> > >>> i is o
> > True
> > 
> > >>> a1 = 'hi'
> > >>> a2 = 'hi'
> > >>> a1 is a2
> > True
> 
> It's more complicated than that:
> 
>     (using Python 2.7.10)
>     >>> i = 2002
>     >>> o = 2002
>     >>> i is o
>     False
>     >>> a = 'what!?'
>     >>> b = 'what!?'
>     >>> a is b
>     False
> 
> A Python implementation can choose when to reuse immutable objects and
> when not to.  Reusing a value has a cost, because the values have to
> be kept, and then found again. So the cost is only paid when there's
> a reasonable chance that the values will actually be needed again.
> And that cost has to be weighed against the opposite cost of simply
> making a new object instead.

(Sorry, forgot an important point) The other reason to reuse objects is
so that equality testing can be shortened with an identity check. Because
strings are (sometimes) reused, CPython can test if two strings are equal
by first checking if they are precisely the same string object, and only
if they are not, by examining individual characters.

String comparison is an important operation that needs to be fast,
especially when you consider the use of strings as keys in dicts. So the
cost of reusing strings is easily worth it because of the time it saves
when doing string comparisons.

> 
> To answer your direct question: floats are so numerous and varied, it's
> reasonable to guess that you won't often need the same one.  And the cost
> to make a new one is so low, that it's simpler just to not reuse any of
> them.

And: floats are rarely checked for equality, and very very very rarely used
as dict keys, so there's no gain by short-circuiting the equality check.

--Ned.


From kwpolska at gmail.com  Fri Nov 25 06:39:03 2016
From: kwpolska at gmail.com (Chris Warrick)
Date: Fri, 25 Nov 2016 12:39:03 +0100
Subject: The Case Against Python 3
In-Reply-To: <o1969f$1p1e$2@gioia.aioe.org>
References: <75df94b8-8af4-4de4-93a8-dd4de5d186b7@googlegroups.com>
 <b44e089f-c7e4-4fd7-8c78-585b8b02269a@googlegroups.com>
 <o1969f$1p1e$2@gioia.aioe.org>
Message-ID: <CAMw+j7KEj=Rkuckv2_hx+MSvb4PNn3so9G6XqeGt8quAJxRSNQ@mail.gmail.com>

On 25 November 2016 at 12:11, Fabien <fabien.maussion at gmail.com> wrote:
> I'd be interested to read what the community thinks about the fact that his
> book (learn the hard way) is extremely influential among beginners, and what
> tools do we have to avoid that beginners stumble across such opinions in
> their very first steps with python...

His book is slowly on the way out. #python on freenode and /r/python
stopped recommending the book. Other places should follow suit, and
actively discourage an outdated (easy_install, distribute, nosetests,
python 2) book written by an asshole and a xenophobe.

-- 
Chris Warrick <https://chriswarrick.com/>
PGP: 5EAAEA16


From bc at freeuk.com  Fri Nov 25 07:16:51 2016
From: bc at freeuk.com (BartC)
Date: Fri, 25 Nov 2016 12:16:51 +0000
Subject: Immutability of Floats, Ints and Strings in Python
In-Reply-To: <31b70ea6-d3ed-4756-81de-e6f327ebb56f@googlegroups.com>
References: <31b70ea6-d3ed-4756-81de-e6f327ebb56f@googlegroups.com>
Message-ID: <o19a1b$mel$1@dont-email.me>

On 25/11/2016 11:24, Nikunj wrote:
>
> Out of curiosity, I wanted to understand the reason behind having different memory location for two identical floats . This is unlike ints or strings. Tried googling but couldn't find anything concrete. Any links or references would be appreciated!

Do you mean for identical floating point literals?

Some implementations will do that; see the test below. Here, Py2 and Py3 
both give the same id for 1.3, but not for a runtime-derived 1.3. Pypy 
will give the same id even for 1.3 at runtime (although I suspect it can 
see past my simple expression and reduce it to 1.3).

Doing this is harder for floats than for integers. So if it is decided 
to have integer values 0 to 1000 as shared objects, then it is easy to 
have a lookup table of 1001 objects so as to avoid repeatedly creating 
and destroying millions of small integers.

But how many floats are there between 0.0 and 1000? Billions I think, 
and the bit-pattern for 1.3 might be 0x3ff4cccccccccccd which doesn't 
lend itself to a simple lookup.

----------------------

print ("Floats:")
x=1.3
y=1.3
print (id(x))
print (id(y))

print (x is y)

z=x+1.3-y

print (id(z))
print (x-z)
print (x==z)
print (x is z)


print ("")
print ("Integers:")
x=13
y=13
print (id(x))
print (id(y))

print (x is y)

z=x+13-y

print (id(z))
print (x-z)
print (x==z)
print (x is z)

-- 
Bartc


From ankur at numeratelabs.com  Fri Nov 25 07:21:18 2016
From: ankur at numeratelabs.com (Ankur Gupta)
Date: Fri, 25 Nov 2016 04:21:18 -0800 (PST)
Subject: Simple Python Quiz To Mark 100th Issue of ImportPython
In-Reply-To: <mailman.5.1480067539.2073.python-list@python.org>
References: <b6cf5b46-406a-42c2-9d18-5ccdd6841d73@googlegroups.com>
 <CAPTjJmpz68+EnYYW5psHb0Qcx_AKb7woqUJdi6z0cf8YfeJvmw@mail.gmail.com>
 <mailman.5.1480067539.2073.python-list@python.org>
Message-ID: <4b11aa8d-dbf2-4700-875e-e34bc7745b26@googlegroups.com>

On Friday, November 25, 2016 at 3:22:30 PM UTC+5:30, Chris Angelico wrote:
> On Fri, Nov 25, 2016 at 8:02 PM, Ankur Gupta <ankur at numeratelabs.com> wrote:
> > Import Python Newsletter completed 2 years and 100 issues. Have a simple fun python quiz http://importpython.com/newsletter/quiz/ to mark the milestone.
> 
> With your question about laying out code, are you aware that PEP 8
> specifically allows both?
> 
> """
> While sometimes it's okay to put an if/for/while with a small body on
> the same line, never do this for multi-clause statements. Also avoid
> folding such long lines!
> """
> 
> You have a very simple condition (actually, trivially simple - in its
> exact form as given, it's always true) guarding a simple statement.
> There's nothing wrong with putting that on a single line. I especially
> consider this important when there are multiple parallel conditions:
> 
> if foo.x: foo.widen()
> if foo.y: foo.heighten()
> if foo.z: foo.deepen()
> 
> There's no point breaking these into two lines each; they show a
> perfect parallel as it is, and splitting the lines would disrupt that.
> 
> ChrisA

Hey ChrisA,

I see merit in what you are saying. I should probably have used an example with multiple statements perhaps to draw a better distinction. My bad. 

Quiz is now out so would be unfair for those who attempted it.

Regards,
Ankur


From ned at nedbatchelder.com  Fri Nov 25 07:35:42 2016
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Fri, 25 Nov 2016 04:35:42 -0800 (PST)
Subject: Immutability of Floats, Ints and Strings in Python
In-Reply-To: <o19a1b$mel$1@dont-email.me>
References: <31b70ea6-d3ed-4756-81de-e6f327ebb56f@googlegroups.com>
 <o19a1b$mel$1@dont-email.me>
Message-ID: <b9df62fb-c0a3-4c67-b44f-b9a5f15ce052@googlegroups.com>

On Friday, November 25, 2016 at 7:17:08 AM UTC-5, BartC wrote:
> On 25/11/2016 11:24, Nikunj wrote:
> >
> > Out of curiosity, I wanted to understand the reason behind having different memory location for two identical floats . This is unlike ints or strings. Tried googling but couldn't find anything concrete. Any links or references would be appreciated!
> 
> Do you mean for identical floating point literals?
> 
> Some implementations will do that; see the test below. Here, Py2 and Py3 
> both give the same id for 1.3, but not for a runtime-derived 1.3. Pypy 
> will give the same id even for 1.3 at runtime (although I suspect it can 
> see past my simple expression and reduce it to 1.3).
>
> ...
> 
> ----------------------
> 
> print ("Floats:")
> x=1.3
> y=1.3
> print (id(x))
> print (id(y))
> 
> print (x is y)
> 
> z=x+1.3-y
> 
> print (id(z))
> print (x-z)
> print (x==z)
> print (x is z)
> 
> 
> print ("")
> print ("Integers:")
> x=13
> y=13
> print (id(x))
> print (id(y))
> 
> print (x is y)
> 
> z=x+13-y
> 
> print (id(z))
> print (x-z)
> print (x==z)
> print (x is z)

Ah, another complication! Literals in code objects are folded together
by the compiler, so you can share objects in one code object even when
they are not generally shared:

    >>> def f():
    ...     a = 1.3
    ...     b = 1.3
    ...     return a, b
    ...
    >>> a, b = f()
    >>> a is b
    True
    >>>
    >>> def g():
    ...     return 1.3
    ...
    >>> def h():
    ...     return 1.3
    ...
    >>> g() is h()
    False
    >>>
    >>> a, b = 1.3, 1.3
    >>> a is b
    True
    >>>
    >>> a = 1.3
    >>> b = 1.3
    >>> a is b
    False

When are objects shared? It's complicated, implementation-dependent,
and can change from version to version. Also, it doesn't matter to
real programs.

--Ned.


From steve+python at pearwood.info  Fri Nov 25 07:44:08 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Fri, 25 Nov 2016 23:44:08 +1100
Subject: Immutability of Floats, Ints and Strings in Python
References: <31b70ea6-d3ed-4756-81de-e6f327ebb56f@googlegroups.com>
 <4630ad9e-e336-4f78-aebb-706d403737ee@googlegroups.com>
 <6d1b57ad-736d-4dc9-8dc3-d835fdb15ebc@googlegroups.com>
Message-ID: <5838321a$0$1596$c3e8da3$5496439d@news.astraweb.com>

On Fri, 25 Nov 2016 10:37 pm, Ned Batchelder wrote:

> And: floats are rarely checked for equality, and very very very rarely
> used as dict keys, so there's no gain by short-circuiting the equality
> check.

You cannot short-circuit the equality check, at least not without giving up
IEEE-754 semantics.

If x is a float, then x == x is not always true.




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From rosuav at gmail.com  Fri Nov 25 08:04:56 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 26 Nov 2016 00:04:56 +1100
Subject: Simple Python Quiz To Mark 100th Issue of ImportPython
In-Reply-To: <4b11aa8d-dbf2-4700-875e-e34bc7745b26@googlegroups.com>
References: <b6cf5b46-406a-42c2-9d18-5ccdd6841d73@googlegroups.com>
 <CAPTjJmpz68+EnYYW5psHb0Qcx_AKb7woqUJdi6z0cf8YfeJvmw@mail.gmail.com>
 <mailman.5.1480067539.2073.python-list@python.org>
 <4b11aa8d-dbf2-4700-875e-e34bc7745b26@googlegroups.com>
Message-ID: <CAPTjJmo86E09K5uQ=1v7X+B00+CWF+Xrtfuxhan=B+OE_THQsQ@mail.gmail.com>

On Fri, Nov 25, 2016 at 11:21 PM, Ankur Gupta <ankur at numeratelabs.com> wrote:
> Hey ChrisA,
>
> I see merit in what you are saying. I should probably have used an example with multiple statements perhaps to draw a better distinction. My bad.
>
> Quiz is now out so would be unfair for those who attempted it.

Fair enough. What you're showing is how hard it is to pin down the
term "Pythonic" - while it's easy to decry some forms of code as being
really bad, there are a lot more that are harder to define, and PEP 8
specifically says you have to look at context.

Congrats on the 100 issue milestone, which I forgot to say in my previous post.

ChrisA


From gengyangcai at gmail.com  Fri Nov 25 08:42:24 2016
From: gengyangcai at gmail.com (Cai Gengyang)
Date: Fri, 25 Nov 2016 21:42:24 +0800
Subject: NameError
In-Reply-To: <CAPjBxg9LrwO3t2VVEs9p+fTL8a6HuCLbyF3rbMAqOU+e=0wNGQ@mail.gmail.com>
References: <223a00e3-1e8b-4ba3-9402-37a7f8ba9dcf@googlegroups.com>
 <CAPM-O+ySA-o3Afh12NU6Ce0REC_pnoH7_we4AOE-NYe9pSqOTw@mail.gmail.com>
 <mailman.77.1479956046.3146.python-list@python.org>
 <079079bb-6abd-4660-be25-dd4f33165e35@googlegroups.com>
 <33bcdbba-9c5d-d884-366b-5897812afbd0@gmx.com>
 <mailman.80.1479960005.3146.python-list@python.org>
 <e4c9f90e-3ef0-41b1-b729-a48792ebada9@googlegroups.com>
 <CAPjBxg9LrwO3t2VVEs9p+fTL8a6HuCLbyF3rbMAqOU+e=0wNGQ@mail.gmail.com>
Message-ID: <CAE8stXxJNOfu2+XTbpkj8yA5Tj7Kupwe_Eag5E8XDYO=UdVOrw@mail.gmail.com>

I can't import it --- keep getting an importerror message. Can you try, let
me know if it works and show me the code ? Thanks alot , appreciate it ...

On Thu, Nov 24, 2016 at 1:22 PM, Nathan Ernst <nathan.ernst at gmail.com>
wrote:

> I don't see anything in that output resembling an error, just a few
> warnings that some features may no be available.
>
> Have you tried importing pygame after you did that? That's what'll prove
> one way or another that it worked.
>
> Regards,
> Nate
>
> On Wed, Nov 23, 2016 at 10:48 PM, Cai Gengyang <gengyangcai at gmail.com>
> wrote:
>
>> Yea, using Mac
>>
>> Following the instructions here for Mac ---https://bitbucket.org/pygam
>> e/pygame/issues/82/homebrew-on-leopard-fails-to-install#comment-627494
>>
>> GengYang Cai CaiGengYangs-MacBook-Pro:~ CaiGengYang$ brew install python
>> ==> Installing dependencies for python: xz, pkg-config, readline, sqlite,
>> ==> Installing python dependency: xz
>> ==> Downloading https://homebrew.bintray.com/.
>> ../xz-5.2.2.yosemite.bottle.ta
>> ########################################################################
>> 100.0%
>> ==> Pouring xz-5.2.2.yosemite.bottle.tar.gz
>> ? /usr/local/Cellar/xz/5.2.2: 91 files, 1.4M
>> ==> Installing python dependency: pkg-config
>> ==> Downloading https://homebrew.bintray.com/.
>> ../pkg-config-0.29.1_1.yosemit
>> ########################################################################
>> 100.0%
>> ==> Pouring pkg-config-0.29.1_1.yosemite.bottle.tar.gz
>> ? /usr/local/Cellar/pkg-config/0.29.1_1: 10 files, 627.3K
>> ==> Installing python dependency: readline
>> ==> Downloading https://homebrew.bintray.com/.
>> ../readline-6.3.8.yosemite.bot
>> ########################################################################
>> 100.0%
>> ==> Pouring readline-6.3.8.yosemite.bottle.tar.gz
>> ==> Caveats
>> This formula is keg-only, which means it was not symlinked into
>> /usr/local.
>>
>> OS X provides the BSD libedit library, which shadows libreadline.
>> In order to prevent conflicts when programs look for libreadline we are
>> defaulting this GNU Readline installation to keg-only.
>>
>> Generally there are no consequences of this for you. If you build your
>> own software and it requires this formula, you'll need to add to your
>> build variables:
>>
>> LDFLAGS: -L/usr/local/opt/readline/lib
>> CPPFLAGS: -I/usr/local/opt/readline/include
>>
>> ==> Summary
>> ? /usr/local/Cellar/readline/6.3.8: 46 files, 2M
>> ==> Installing python dependency: sqlite
>> ==> Downloading https://homebrew.bintray.com/.
>> ../sqlite-3.13.0.yosemite.bott
>> ########################################################################
>> 100.0%
>> ==> Pouring sqlite-3.13.0.yosemite.bottle.tar.gz
>> ==> Caveats
>> This formula is keg-only, which means it was not symlinked into
>> /usr/local.
>>
>> OS X provides an older sqlite3.
>>
>> Generally there are no consequences of this for you. If you build your
>> own software and it requires this formula, you'll need to add to your
>> build variables:
>>
>> LDFLAGS: -L/usr/local/opt/sqlite/lib
>> CPPFLAGS: -I/usr/local/opt/sqlite/include
>>
>> ==> Summary
>> ? /usr/local/Cellar/sqlite/3.13.0: 10 files, 2.9M
>> ==> Installing python dependency: gdbm
>> ==> Downloading https://homebrew.bintray.com/.
>> ../gdbm-1.12.yosemite.bottle.t
>> ########################################################################
>> 100.0%
>> ==> Pouring gdbm-1.12.yosemite.bottle.tar.gz
>> ? /usr/local/Cellar/gdbm/1.12: 18 files, 490.8K
>> ==> Installing python dependency: openssl
>> ==> Downloading https://homebrew.bintray.com/.
>> ../openssl-1.0.2h_1.yosemite.b
>> ########################################################################
>> 100.0%
>> ==> Pouring openssl-1.0.2h_1.yosemite.bottle.tar.gz
>> ==> Caveats
>> A CA file has been bootstrapped using certificates from the system
>> keychain. To add additional certificates, place .pem files in
>> /usr/local/etc/openssl/certs
>>
>> and run
>> /usr/local/opt/openssl/bin/c_rehash
>>
>> This formula is keg-only, which means it was not symlinked into
>> /usr/local.
>>
>> Apple has deprecated use of OpenSSL in favor of its own TLS and crypto
>> libraries
>>
>> Generally there are no consequences of this for you. If you build your
>> own software and it requires this formula, you'll need to add to your
>> build variables:
>>
>> LDFLAGS: -L/usr/local/opt/openssl/lib
>> CPPFLAGS: -I/usr/local/opt/openssl/include
>>
>> ==> Summary
>> ? /usr/local/Cellar/openssl/1.0.2h_1: 1,691 files, 12.0M
>> ==> Installing python
>> Warning: Building python from source:
>> The bottle needs the Apple Command Line Tools to be installed.
>> You can install them, if desired, with:
>> xcode-select --install
>>
>> ==> Downloading https://www.python.org/.../2.7.12/Python-2.7.12.tar.xz
>> ########################################################################
>> 100.0%
>> ==> Downloading https://bugs.python.org/file30
>> 805/issue10910-workaround.txt
>> ########################################################################
>> 100.0%
>> ==> Patching
>> ==> Applying issue10910-workaround.txt
>> patching file Include/pyport.h
>> Hunk #1 succeeded at 713 (offset 14 lines).
>> Hunk #2 succeeded at 736 (offset 14 lines).
>> ==> ./configure --prefix=/usr/local/Cellar/python/2.7.12 --enable-ipv6
>> --dataroo
>> ==> make
>> /usr/local/share/python/easy_install mecurial
>>
>> brew install sdl
>> brew install sdl_mixer
>> brew install sdl_ttf
>> brew install sdl_image
>>
>> hg clone https://bitbucket.org/pygame/pygame
>> cd pygame
>> /usr/local/bin/python setup.py install
>>
>>
>> Does this work ?
>>
>>
>>
>>
>>
>>
>>
>> On Thursday, November 24, 2016 at 12:00:18 PM UTC+8, Thomas Nyberg wrote:
>> > On 11/23/2016 10:02 PM, Cai Gengyang wrote:
>> > > I tried to import pygame by using these commands
>> ------------------------------------------------https://www.
>> google.com.sg/#q=how+to+import+pygame
>> > >
>> > > but this is the error I got :
>> > >
>> > > CaiGengYangs-MacBook-Pro:~ CaiGengYang$ sudo apt-get install
>> python-pygame
>> > > sudo: apt-get: command not found
>> > >
>> > Judging by your prompt it looks like you're on a Mac. The apt-get
>> > program is installed on some Linux distributions (probably your
>> > instructions are for Ubuntu). Here are some istructions for installing
>> > pygame for a Macbook:
>> >
>> >       http://pygame.org/wiki/macintosh
>> >       http://florian-berger.de/en/articles/installing-pygame-for-
>> python-3-on-os-x/
>> >       https://jamesfriend.com.au/installing-pygame-python-3-mac-
>> os-yosemite
>> >
>> > Here's some info about getting a package manager like apt for Mac:
>> >
>> >       http://unix.stackexchange.com/questions/80711/how-to-ins
>> tall-apt-get-or-yum-on-mac-os-x
>> >
>> > Hopefully this helps.
>> >
>> > Cheers,
>> > Thomas
>> --
>> https://mail.python.org/mailman/listinfo/python-list
>>
>
>


From ankur at numeratelabs.com  Fri Nov 25 09:33:04 2016
From: ankur at numeratelabs.com (Ankur Gupta)
Date: Fri, 25 Nov 2016 06:33:04 -0800 (PST)
Subject: Simple Python Quiz To Mark 100th Issue of ImportPython
In-Reply-To: <mailman.9.1480079098.2073.python-list@python.org>
References: <b6cf5b46-406a-42c2-9d18-5ccdd6841d73@googlegroups.com>
 <CAPTjJmpz68+EnYYW5psHb0Qcx_AKb7woqUJdi6z0cf8YfeJvmw@mail.gmail.com>
 <mailman.5.1480067539.2073.python-list@python.org>
 <4b11aa8d-dbf2-4700-875e-e34bc7745b26@googlegroups.com>
 <CAPTjJmo86E09K5uQ=1v7X+B00+CWF+Xrtfuxhan=B+OE_THQsQ@mail.gmail.com>
 <mailman.9.1480079098.2073.python-list@python.org>
Message-ID: <1297c58b-84fb-48dd-81de-577fe1a64416@googlegroups.com>

On Friday, November 25, 2016 at 6:35:13 PM UTC+5:30, Chris Angelico wrote:
> On Fri, Nov 25, 2016 at 11:21 PM, Ankur Gupta <ankur at numeratelabs.com> wrote:
> > Hey ChrisA,
> >
> > I see merit in what you are saying. I should probably have used an example with multiple statements perhaps to draw a better distinction. My bad.
> >
> > Quiz is now out so would be unfair for those who attempted it.
> 
> Fair enough. What you're showing is how hard it is to pin down the
> term "Pythonic" - while it's easy to decry some forms of code as being
> really bad, there are a lot more that are harder to define, and PEP 8
> specifically says you have to look at context.
> 
> Congrats on the 100 issue milestone, which I forgot to say in my previous post.
> 
> ChrisA

Thanks Chris :) 


From hemla21 at gmail.com  Fri Nov 25 10:17:39 2016
From: hemla21 at gmail.com (Heli)
Date: Fri, 25 Nov 2016 07:17:39 -0800 (PST)
Subject: best way to read a huge ascii file.
Message-ID: <eeed7603-4a21-4f7d-a484-5962bf44a30d@googlegroups.com>

Hi, 

I have a huge ascii file(40G) and I have around 100M lines.  I read this file using :

f=np.loadtxt(os.path.join(dir,myfile),delimiter=None,skiprows=0) 

x=f1[:,1]
y=f1[:,2]
z=f1[:,3]
id=f1[:,0]

I will need the x,y,z and id arrays later for interpolations. The problem is reading the file takes around 80 min while the interpolation only takes 15 mins. 

I was wondering if there is a more optimized way to read the file that would  reduce the time to read the input file?

I have the same problem when writing the output using np.savetxt. 

Thank you in Advance for your help, 


From marko at pacujo.net  Fri Nov 25 10:29:44 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Fri, 25 Nov 2016 17:29:44 +0200
Subject: best way to read a huge ascii file.
References: <eeed7603-4a21-4f7d-a484-5962bf44a30d@googlegroups.com>
Message-ID: <87inrbpogn.fsf@elektro.pacujo.net>

Heli <hemla21 at gmail.com>:

> I have a huge ascii file(40G) and I have around 100M lines. I read this
> file using :
>
> [...]
>
> The problem is reading the file takes around 80 min while the
> interpolation only takes 15 mins.
>
> I was wondering if there is a more optimized way to read the file that
> would reduce the time to read the input file?

It looks pretty fast to me already; that's 20,000 lines per second, or
65 megabits per second.


Marko


From nathan.ernst at gmail.com  Fri Nov 25 11:23:46 2016
From: nathan.ernst at gmail.com (Nathan Ernst)
Date: Fri, 25 Nov 2016 10:23:46 -0600
Subject: printing funny symbols within spyder ide
In-Reply-To: <e872f$583743b2$54817cc5$2721@news1.surfino.com>
References: <e872f$583743b2$54817cc5$2721@news1.surfino.com>
Message-ID: <CAPjBxg_FfPXqvn+rOrEQNQo_sw294Vf+mN4pxUT9KL+8=UcQ0w@mail.gmail.com>

You're attempting to print out control characters most of which have no
visible representation. For "\7", at least if you're running from bash, and
not in an IDE, you should get an audible bell. All decimal ordinals below
32 are control

You can find a list of the symbols here:
http://en.cppreference.com/w/c/language/ascii

Even though this is an ASCII table, the ordinals for characters are the
same as in UTF-8 encoding.

Regards.

On Thu, Nov 24, 2016 at 1:46 PM, andy <wants at no.mail.local> wrote:

> when printing these 'escaped-number-texts' within Linux/Spyder3 ide
> console, i get funny symbols like a "phone-symbol", triangles or symbols
> for male or female.
>
> >>> print("\7") # gives a phone
> >>> print("\5")
> >>> print("\1")
> >>> print("\21")
> >>> print("\30")
> >>> print("\31")
> >>> print("\32")
>
> def phonesymbol():
>         print("\7")
>
> phonesymbol() # prints a phone symbol
>
> my question is, is this a bug or a feature of Spyder3?
> if it is a feature, where can I find the complete table of symbols?
>
> within bash - starting python3 - i get some ugly "unicode?" symbols,
> which look like a square containig tiny alphanumeric numbers.
>
> best regards
> andy
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From ned at nedbatchelder.com  Fri Nov 25 11:25:55 2016
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Fri, 25 Nov 2016 08:25:55 -0800 (PST)
Subject: A Rebuttal For Python 3
In-Reply-To: <cbe0720c-d54a-4682-b3c5-5206f5cff3a1@googlegroups.com>
References: <3aaa8c9b-6f25-4a40-8378-aa51a6da352e@googlegroups.com>
 <cbe0720c-d54a-4682-b3c5-5206f5cff3a1@googlegroups.com>
Message-ID: <fb7acaa8-c19a-4100-88a6-80b56f845196@googlegroups.com>

On Friday, November 25, 2016 at 3:45:37 AM UTC-5, Rustom Mody wrote:
> On Friday, November 25, 2016 at 1:27:18 AM UTC+5:30, bream... at gmail.com wrote:
> > https://eev.ee/blog/2016/11/23/a-rebuttal-for-python-3/ is one of presumably many responses to the article I posted about under the subject "The Case Against Python 3".  Enjoy :)
> 
> I'd say we are living in astrological? times?
> 
> There was a recent election?
> In a grand and famous nation
> The expletives xchanged
> Midst ppl deranged
> Let someone else complete the ditty
> 
> Considering that the last lines of the article usually contain the summary of the article.  And they run as follows:
> 
> |Fuck off...
> |
> | This is because you don?t know what you?re talking about, and you should shut up.
> 
> I am just going to comply (unless someone explain why so much uncivility is such a great idea)


I am not a fan of incivility, but this piece is a bit unusual. As eevee
explained in his rebuttal, Zed's rant is wildly irresponsible, both on a 
technical level, because a number of his claims are simply wrong; and on
a cultural level, because his audience is beginners looking to learn Python,
who will not have the technical skill to understand how misleading and 
irresponsible his arguments are.

On a second level, Zed has made a name for himself online as a controversial
figure, using abusive language and stances to gain attention.  His piece
here is actually quite calm compared to some of his writing.  But I'm OK
with eevee using Zed's techniques against him, especially after calmly
and logically dismantling the arguments.

--Ned.
against Python 3 is


From ned at nedbatchelder.com  Fri Nov 25 11:36:14 2016
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Fri, 25 Nov 2016 08:36:14 -0800 (PST)
Subject: A Rebuttal For Python 3
In-Reply-To: <fb7acaa8-c19a-4100-88a6-80b56f845196@googlegroups.com>
References: <3aaa8c9b-6f25-4a40-8378-aa51a6da352e@googlegroups.com>
 <cbe0720c-d54a-4682-b3c5-5206f5cff3a1@googlegroups.com>
 <fb7acaa8-c19a-4100-88a6-80b56f845196@googlegroups.com>
Message-ID: <76a252ef-4b67-4d5e-b26b-d3b445d0efa1@googlegroups.com>

On Friday, November 25, 2016 at 11:26:09 AM UTC-5, Ned Batchelder wrote:
> 
> --Ned.
> against Python 3 is

Umm, no idea where that "signature" came from... I am not against
Python 3... :)


From bc at freeuk.com  Fri Nov 25 11:39:12 2016
From: bc at freeuk.com (BartC)
Date: Fri, 25 Nov 2016 16:39:12 +0000
Subject: best way to read a huge ascii file.
In-Reply-To: <eeed7603-4a21-4f7d-a484-5962bf44a30d@googlegroups.com>
References: <eeed7603-4a21-4f7d-a484-5962bf44a30d@googlegroups.com>
Message-ID: <o19pd8$ha6$1@dont-email.me>

On 25/11/2016 15:17, Heli wrote:

> I have a huge ascii file(40G) and I have around 100M lines.  I read this file using :
>
> f=np.loadtxt(os.path.join(dir,myfile),delimiter=None,skiprows=0)
>
> x=f1[:,1]
> y=f1[:,2]
> z=f1[:,3]
> id=f1[:,0]
>
> I will need the x,y,z and id arrays later for interpolations. The problem is reading the file takes around 80 min while the interpolation only takes 15 mins.
>
> I was wondering if there is a more optimized way to read the file that would  reduce the time to read the input file?
>
> I have the same problem when writing the output using np.savetxt.

Is that read entirely into RAM? I suppose lines are discarded once they 
are read otherwise it would have to load 40GB before it can do anything.

How much of your RAM is used up during the operation? If that starts to 
get full then it can get very slow. (Same with a fragmented hard drive.)

Where does the file come from (with savetxt?); could it be generated 
more compactly?

I don't quite understand what f1[:,1] does, but if that's a slice, that 
usually involves copying (and extra memory). But presumably you've 
already measured the 80 minutes just to do the np.loadtxt part. (I guess 
the processing won't allow the file to be split into smaller pieces.)


-- 
Bartc



From amirouche.boubekki at gmail.com  Fri Nov 25 12:30:37 2016
From: amirouche.boubekki at gmail.com (Amirouche Boubekki)
Date: Fri, 25 Nov 2016 17:30:37 +0000
Subject: Anyone needs a graphdb written in Python?
In-Reply-To: <CAL7_Mo_ANsYGv=qcDgwxEM4hKArPTQj68C2z1SJk33O4YJig9g@mail.gmail.com>
References: <CAL7_Mo_ANsYGv=qcDgwxEM4hKArPTQj68C2z1SJk33O4YJig9g@mail.gmail.com>
Message-ID: <CAL7_Mo9hO0tZrjtmQE6EkOXmB1Tti=-Z4Sx0KAV_+VGae-m90g@mail.gmail.com>

H?llo again,

On Tue, Nov 22, 2016 at 9:56 PM Amirouche Boubekki <
amirouche.boubekki at gmail.com> wrote:

> I am working on a graphdb written Python on top of wiredtiger.
>
> Anyone want to share about the subject about where this could be made
> useful?
>

A bit of context might be helpful.

I've been working for sometime a graph database library on top wiredtiger
called AjguDB <https://github.com/amirouche/AjguDB>.

Now I am happy with the code and would like get to the next level and show
that this tool can be helpful in situation where bigger than RAM graph data
is used.

I could simply adapt networkx algorithm to my graph API but this sound
boring instead what would like to do is solve a real problem.

Do you know such problems, if possible related to wikidata or text mining.

Thanks in advance!


From greg.ewing at canterbury.ac.nz  Fri Nov 25 17:17:52 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Sat, 26 Nov 2016 11:17:52 +1300
Subject: The Case Against Python 3
In-Reply-To: <mailman.4.1480066279.2073.python-list@python.org>
References: <75df94b8-8af4-4de4-93a8-dd4de5d186b7@googlegroups.com>
 <b44e089f-c7e4-4fd7-8c78-585b8b02269a@googlegroups.com>
 <CAPTjJmrQyp=Bs2MmMM9+5aUf54XmgydcMADnODJ65USUD0GwRA@mail.gmail.com>
 <mailman.4.1480066279.2073.python-list@python.org>
Message-ID: <e9rrkiFk2r8U1@mid.individual.net>

Chris Angelico wrote:
> but brace-formatting lets you reorder the parameters, so it has
> flexibility that can be important for i18n.

Actually, Python's version of %-formatting lets you reorder
the parameters as well. The brace syntax for this is easier
to read and write, though, so probably better for i18n files.

-- 
Greg


From steve+python at pearwood.info  Fri Nov 25 17:36:17 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sat, 26 Nov 2016 09:36:17 +1100
Subject: best way to read a huge ascii file.
References: <eeed7603-4a21-4f7d-a484-5962bf44a30d@googlegroups.com>
Message-ID: <5838bce2$0$22142$c3e8da3$5496439d@news.astraweb.com>

On Sat, 26 Nov 2016 02:17 am, Heli wrote:

> Hi,
> 
> I have a huge ascii file(40G) and I have around 100M lines.  I read this
> file using :
> 
> f=np.loadtxt(os.path.join(dir,myfile),delimiter=None,skiprows=0)
[...]
> I will need the x,y,z and id arrays later for interpolations. The problem
> is reading the file takes around 80 min while the interpolation only takes
> 15 mins.

There's no way of telling whether this is good performance or bad
performance. Where are you reading it from? Over a network file share on
the other side of the world? From a USB hard drive with a USB 2 cable? From
a blazing fast SSD hard drive running on a server-class machine with a TB
of RAM?

My suggestion is that before you spend any more time trying to optimize the
software, you try a simple test that will tell you whether or not you are
wasting your time. Try making a copy of this 40GB file. I'd expect that
making a copy should take *longer* than just reading the file, because you
have to read and write 40GB. If you find that it takes (let's say) 30
minutes to read and write a copy of the file, and 80 minutes for numpy to
read the file, then its worth looking at optimizing the process. But if it
takes (say) 200 minutes to read and write the file, then probably not. When
you're dealing with large quantities of data, it takes time to move that
many bytes from place to place.

It would also help if you told us a bit more about the machine you are
running on, specifically how much RAM do you have. If you're trying to
process a 40GB file in memory on a machine with 2GB of RAM, you're going to
have a bad time...


-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From jsf80238 at gmail.com  Fri Nov 25 21:03:08 2016
From: jsf80238 at gmail.com (Jason Friedman)
Date: Fri, 25 Nov 2016 19:03:08 -0700
Subject: Options for stdin and stdout when using pdb debugger
In-Reply-To: <baaf02f9-7c6c-4450-9e87-77ddacb5dcaa@googlegroups.com>
References: <baaf02f9-7c6c-4450-9e87-77ddacb5dcaa@googlegroups.com>
Message-ID: <CANy1k1gcnMrb1RL4jPYPab=yg5Xh9mJDAdhDAAB+WWpbPrTzrA@mail.gmail.com>

> I would like to use pdb in an application where it isn't possible to use sys.stdin for input. I've read in the documentation for pdb.Pdb that a file object can be used instead of sys.stdin. Unfortunately, I'm not clear about my options for the file object.
>
> I've looked at rpdb on PyPI, which re-routes stdin and stdout to a socket handler. I can connect to the socket with telnet. This works well.

You can use the code from
https://docs.python.org/3/library/fileinput.html#module-fileinput to
change this code:

for line in sys.stdin.readlines():
    process(line)

to this:

import fileinput
for line in fileinput.input():
    process(line)

Then, write the data you would expect to come on STDIN to a file and
run your program:

$ python my-program.py myfile

It will work as normal when you run it normally:

$ data-generation-command | python my-program.py


From mittra at juno.com  Fri Nov 25 22:33:11 2016
From: mittra at juno.com (qrious)
Date: Fri, 25 Nov 2016 19:33:11 -0800 (PST)
Subject: Python turtle: How to change icons
Message-ID: <982f08af-473e-4361-8e89-2b1500425648@googlegroups.com>


Hello All, 

I would like to change two graphical icons related to turtle graphics using Python:

a) One that shows up at the top left corner of the canvas window as in below. I believe this is coming from tk itself.

https://s22.postimg.org/tkjaxmh41/image.png 

b) The icon on the desktop as in below (in Windows): 

https://s13.postimg.org/n7ol0mdpz/icon.png 

Can this be done from within python code itself?

Thanks very much. 


From tjreedy at udel.edu  Sat Nov 26 02:59:12 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 26 Nov 2016 02:59:12 -0500
Subject: Python turtle: How to change icons
In-Reply-To: <982f08af-473e-4361-8e89-2b1500425648@googlegroups.com>
References: <982f08af-473e-4361-8e89-2b1500425648@googlegroups.com>
Message-ID: <o1bfcf$njq$1@blaine.gmane.org>

On 11/25/2016 10:33 PM, qrious wrote:
>
> Hello All,
>
> I would like to change two graphical icons related to turtle graphics using Python:
>
> a) One that shows up at the top left corner of the canvas window as in below. I believe this is coming from tk itself.
>
> https://s22.postimg.org/tkjaxmh41/image.png

That can be changed for app after creating root window and IDLE does so. 
  See idlelib/pyshell.py (in 3.6, PyShell.py previously), after "# set 
application icon".  Code is different for Windows and *nix/Mac.

> b) The icon on the desktop as in below (in Windows):
>
> https://s13.postimg.org/n7ol0mdpz/icon.png

This is set during installation.  I *presume* (but know no details) that 
icon file is loaded somewhat and appropriate registry setting connects 
app to icon file.


-- 
Terry Jan Reedy



From ian.g.kelly at gmail.com  Sat Nov 26 03:01:44 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Sat, 26 Nov 2016 01:01:44 -0700
Subject: The Case Against Python 3
In-Reply-To: <b44e089f-c7e4-4fd7-8c78-585b8b02269a@googlegroups.com>
References: <75df94b8-8af4-4de4-93a8-dd4de5d186b7@googlegroups.com>
 <b44e089f-c7e4-4fd7-8c78-585b8b02269a@googlegroups.com>
Message-ID: <CALwzidnCSVg62MR8Q_nA02Pindci9279WajKnS3HBw3Ow4BkCA@mail.gmail.com>

On Fri, Nov 25, 2016 at 1:29 AM, Mark Summerfield <list at qtrac.plus.com> wrote:
> The article has a section called:
>
>     "Too Many Formatting Options"
>
> He's right! The % formatting was kept to help port old code, the new .format() which is far more versatile is a bit verbose, so finally they've settled on f-strings. So, you do need to know that all three exist (e.g., for maintaining code), but you can easily choose the style that you prefer and just use that.

Largely tangential, but don't you mean "all four"? PEP-292 template
strings are also still around, though I don't recall that I've ever
seen them used in the wild.

When I read that Python 3.6 would include f-strings, I turned to the
coworker sitting next to me and said, "Oh my god, Python is adding yet
another new syntax for string formatting." It's getting to be a joke.


From python.list at tim.thechases.com  Sat Nov 26 08:17:11 2016
From: python.list at tim.thechases.com (Tim Chase)
Date: Sat, 26 Nov 2016 07:17:11 -0600
Subject: The Case Against Python 3
In-Reply-To: <CALwzidnCSVg62MR8Q_nA02Pindci9279WajKnS3HBw3Ow4BkCA@mail.gmail.com>
References: <75df94b8-8af4-4de4-93a8-dd4de5d186b7@googlegroups.com>
 <b44e089f-c7e4-4fd7-8c78-585b8b02269a@googlegroups.com>
 <CALwzidnCSVg62MR8Q_nA02Pindci9279WajKnS3HBw3Ow4BkCA@mail.gmail.com>
Message-ID: <20161126071711.57aa22a1@bigbox.christie.dr>

On 2016-11-26 01:01, Ian Kelly wrote:
> When I read that Python 3.6 would include f-strings, I turned to the
> coworker sitting next to me and said, "Oh my god, Python is adding
> yet another new syntax for string formatting." It's getting to be a
> joke.

Pretty soon Python will have one string-formatting syntax for every
Python web framework. ;-)

-tkc




From junkone1 at gmail.com  Sat Nov 26 11:12:46 2016
From: junkone1 at gmail.com (junkone1 at gmail.com)
Date: Sat, 26 Nov 2016 08:12:46 -0800 (PST)
Subject: how do i fix this invalid arguement error
Message-ID: <2cb245b2-98c3-41e7-8c01-0ece703fc72d@googlegroups.com>

import csv
with open('\\192.168.0.1\fe18cb0618cabd41\ninjatrader$EURUSDTestRun 2016-11-25-11-11.csv','r') as f:
    reader=csv.reader(f)
    for row in reader:
        print(row)


  File "C:/Users/Suresh/PycharmProjects/untitled/test.py", line 2, in <module>
    with open('\\192.168.0.1\fe18cb0618cabd41\ninjatrader$EURUSDTestRun 2016-11-25-11-11.csv','r') as f:
OSError: [Errno 22] Invalid argument: '\\192.168.0.1\x0ce18cb0618cabd41\ninjatrader$EURUSDTestRun 2016-11-25-11-11.csv'


From tjreedy at udel.edu  Sat Nov 26 14:18:32 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 26 Nov 2016 14:18:32 -0500
Subject: how do i fix this invalid arguement error
In-Reply-To: <amij3chi5llhsngv84jpc8e3g04hdv8qq0@4ax.com>
References: <2cb245b2-98c3-41e7-8c01-0ece703fc72d@googlegroups.com>
 <amij3chi5llhsngv84jpc8e3g04hdv8qq0@4ax.com>
Message-ID: <o1cn65$7kr$1@blaine.gmane.org>

On 11/26/2016 12:55 PM, Dennis Lee Bieber wrote:
> On Sat, 26 Nov 2016 08:12:46 -0800 (PST), junkone1 at gmail.com declaimed the
> following:
>
>> import csv
>> with open('\\192.168.0.1\fe18cb0618cabd41\ninjatrader$EURUSDTestRun 2016-11-25-11-11.csv','r') as f:
>>    reader=csv.reader(f)
>>    for row in reader:
>>        print(row)
>>
>>
>>  File "C:/Users/Suresh/PycharmProjects/untitled/test.py", line 2, in <module>
>>    with open('\\192.168.0.1\fe18cb0618cabd41\ninjatrader$EURUSDTestRun 2016-11-25-11-11.csv','r') as f:
>> OSError: [Errno 22] Invalid argument: '\\192.168.0.1\x0ce18cb0618cabd41\ninjatrader$EURUSDTestRun 2016-11-25-11-11.csv'
>
>
> 	First suggestion... Make it a RAW string. What you have has an embedded
> new-line character making it:
> \\192.168.0.1\fe18cb0618cabd41
> injatrader$EURUSDTestRun 2016-11-25-11-11.csv

Or use forward slashes:
with open('/192.168.0.1/fe18cb0618cabd41/ninjatrader$EURUSDTestRun 
2016-11-25-11-11.csv','r') as f:

> 	Second... does Python open() accept web addresses? What is at
> 192.168.0.1 that you don't have some more sensible path to it? 192.168.0.1
> is a private local network address. You are on a Windows machine so if the
> file is available on a local network node you can probably mount that node
> (Computer/Map Network Drive) and access the file as something like
> Z:/ninjatrader...
>


-- 
Terry Jan Reedy



From steve+python at pearwood.info  Sat Nov 26 19:06:48 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sun, 27 Nov 2016 11:06:48 +1100
Subject: how do i fix this invalid arguement error
References: <2cb245b2-98c3-41e7-8c01-0ece703fc72d@googlegroups.com>
Message-ID: <583a239a$0$1596$c3e8da3$5496439d@news.astraweb.com>

On Sun, 27 Nov 2016 03:12 am, junkone1 at gmail.com wrote:

> import csv
> with open('\\192.168.0.1\fe18cb0618cabd41\ninjatrader$EURUSDTestRun
> 2016-11-25-11-11.csv','r') as f:

\f inserts a FORMFEED character, ASCII code 12. \n inserts a LINEFEED
character, or newline, ASCII code 10. I believe that on Windows, both are
illegal in file names. And \\ inserts a single backslash.

You have three solutions:

(1) Use forward slashes, which Windows accepts as well:

'//192.168.0.1/fe18cb0618cabd41/ninjatrader$EURUSDTestRun
2016-11-25-11-11.csv'


(2) Use a "raw string" to disable most (but not all) backslash escapes:

r'\\192.168.0.1\fe18cb0618cabd41\ninjatrader$EURUSDTestRun
2016-11-25-11-11.csv'


The problem with raw strings is that you cannot end a string with a
backslash, so you cannot write:

# this gives a syntax error
r'C:\My Documents\directory\'


(3) Or escape all your backslashes with more backslashes:

'\\\\192.168.0.1\\fe18cb0618cabd41\\ninjatrader$EURUSDTestRun
2016-11-25-11-11.csv'




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Sat Nov 26 19:13:25 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sun, 27 Nov 2016 11:13:25 +1100
Subject: The Case Against Python 3
References: <75df94b8-8af4-4de4-93a8-dd4de5d186b7@googlegroups.com>
 <b44e089f-c7e4-4fd7-8c78-585b8b02269a@googlegroups.com>
 <CALwzidnCSVg62MR8Q_nA02Pindci9279WajKnS3HBw3Ow4BkCA@mail.gmail.com>
 <mailman.17.1480147352.2073.python-list@python.org>
Message-ID: <583a2527$0$22140$c3e8da3$5496439d@news.astraweb.com>

On Sat, 26 Nov 2016 07:01 pm, Ian Kelly wrote:

> When I read that Python 3.6 would include f-strings, I turned to the
> coworker sitting next to me and said, "Oh my god, Python is adding yet
> another new syntax for string formatting." It's getting to be a joke.

f-strings are not merely string formatting. They are a new syntax for
evaluating arbitrary Python expressions, which then gets inserted into a
string.

In some ways, they're like the old Python 2 backtick syntax:

py> `1 + len(str(5**4))`
'4'


except that you can automagically concatenate strings to the evaluated
expressions.

So-called f-strings haven't even hit the  already been implicated in a
code-injection vulnerability:

http://bugs.python.org/issue28563

I feel kind of vindicated here, because when so-called f-strings were first
proposed I asked about the security implication of another way of
evaluating arbitrary expressions, and I was told that there were no
security implications. Technically that might be true in the sense that
f-strings don't do anything that wasn't already possible, but as the above
bug shows, they can make exploiting code injection trivially easy in cases
where they were previously diabolically hard.

Yay for progress.


-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From rosuav at gmail.com  Sat Nov 26 19:25:06 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 27 Nov 2016 11:25:06 +1100
Subject: The Case Against Python 3
In-Reply-To: <583a2527$0$22140$c3e8da3$5496439d@news.astraweb.com>
References: <75df94b8-8af4-4de4-93a8-dd4de5d186b7@googlegroups.com>
 <b44e089f-c7e4-4fd7-8c78-585b8b02269a@googlegroups.com>
 <CALwzidnCSVg62MR8Q_nA02Pindci9279WajKnS3HBw3Ow4BkCA@mail.gmail.com>
 <mailman.17.1480147352.2073.python-list@python.org>
 <583a2527$0$22140$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPTjJmoYSS-FLsF8e_Gs_eeb3AeF2jKgUe9ZLGwpSw5KQyay5A@mail.gmail.com>

On Sun, Nov 27, 2016 at 11:13 AM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> So-called f-strings haven't even hit the  already been implicated in a
> code-injection vulnerability:
>
> http://bugs.python.org/issue28563
>
> I feel kind of vindicated here, because when so-called f-strings were first
> proposed I asked about the security implication of another way of
> evaluating arbitrary expressions, and I was told that there were no
> security implications. Technically that might be true in the sense that
> f-strings don't do anything that wasn't already possible, but as the above
> bug shows, they can make exploiting code injection trivially easy in cases
> where they were previously diabolically hard.

Given that the exploit exists in 2.7, I would say f-strings didn't
create this, eval did. The problem is that you absolutely CANNOT
"sanitize" something before giving it to eval. An f-string slips past
the sanitizer, but so do other things.

ChrisA


From steve+python at pearwood.info  Sat Nov 26 20:19:08 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Sun, 27 Nov 2016 12:19:08 +1100
Subject: The Case Against Python 3
References: <75df94b8-8af4-4de4-93a8-dd4de5d186b7@googlegroups.com>
 <b44e089f-c7e4-4fd7-8c78-585b8b02269a@googlegroups.com>
 <CALwzidnCSVg62MR8Q_nA02Pindci9279WajKnS3HBw3Ow4BkCA@mail.gmail.com>
 <mailman.17.1480147352.2073.python-list@python.org>
 <583a2527$0$22140$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmoYSS-FLsF8e_Gs_eeb3AeF2jKgUe9ZLGwpSw5KQyay5A@mail.gmail.com>
 <mailman.26.1480206315.2073.python-list@python.org>
Message-ID: <583a348e$0$1617$c3e8da3$5496439d@news.astraweb.com>

On Sun, 27 Nov 2016 11:25 am, Chris Angelico wrote:

> On Sun, Nov 27, 2016 at 11:13 AM, Steve D'Aprano
> <steve+python at pearwood.info> wrote:
>> So-called f-strings haven't even hit the  already been implicated in a
>> code-injection vulnerability:
>>
>> http://bugs.python.org/issue28563
>>
>> I feel kind of vindicated here, because when so-called f-strings were
>> first proposed I asked about the security implication of another way of
>> evaluating arbitrary expressions, and I was told that there were no
>> security implications. Technically that might be true in the sense that
>> f-strings don't do anything that wasn't already possible, but as the
>> above bug shows, they can make exploiting code injection trivially easy
>> in cases where they were previously diabolically hard.
> 
> Given that the exploit exists in 2.7, I would say f-strings didn't
> create this, eval did. 

I never said that f-strings caused the vulnerability. I choose my words
carefully. As I said when I mentioned this issue three weeks ago, the
underlying cause of the vulnerability is the use of eval on an untrusted
string. But the existence of a theoretical vulnerability is not the same as
an exploit, let alone an easy exploit.


> The problem is that you absolutely CANNOT 
> "sanitize" something before giving it to eval.

Be careful about making absolute claims. I challenge you to break this use
of eval:

def calculate(phrase):
    try:
        phrase = sanitize(phrase)
    except ValueError:
        return
    return eval(phrase, {'x': 20})

    
def sanitize(phrase):
    phrase = phrase.replace(' ', '')
    if phrase in ('x+1', '2*x'):
        return phrase
    raise ValueError('unsafe phrase')


For a more practical example, namedtuple uses exec to dynamically build the
class. Can you find a code injection attack in namedtuple? I doubt it. Not
all uses of exec or eval lead to a code injection vulnerability.


> An f-string slips past the sanitizer, but so do other things.

I daresay you are right that a sufficiently clever adversary may have found
an exploit. But there's no sign that anyone actually did find an exploit,
until f-strings made exploiting this trivial.





-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From nathan.ernst at gmail.com  Sat Nov 26 20:26:00 2016
From: nathan.ernst at gmail.com (Nathan Ernst)
Date: Sat, 26 Nov 2016 19:26:00 -0600
Subject: The Case Against Python 3
In-Reply-To: <583a348e$0$1617$c3e8da3$5496439d@news.astraweb.com>
References: <75df94b8-8af4-4de4-93a8-dd4de5d186b7@googlegroups.com>
 <b44e089f-c7e4-4fd7-8c78-585b8b02269a@googlegroups.com>
 <CALwzidnCSVg62MR8Q_nA02Pindci9279WajKnS3HBw3Ow4BkCA@mail.gmail.com>
 <mailman.17.1480147352.2073.python-list@python.org>
 <583a2527$0$22140$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmoYSS-FLsF8e_Gs_eeb3AeF2jKgUe9ZLGwpSw5KQyay5A@mail.gmail.com>
 <mailman.26.1480206315.2073.python-list@python.org>
 <583a348e$0$1617$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPjBxg-FEHwDzP7_+Bp82dKkY9cJ8aoMZWMMfVrWW0iHDbf0Vg@mail.gmail.com>

Sure, what if the input used a double quote instead of single, cursory
glance looks like it might vulnerable.

(Not trying to be argumentative here)

On Nov 26, 2016 7:21 PM, "Steve D'Aprano" <steve+python at pearwood.info>
wrote:

> On Sun, 27 Nov 2016 11:25 am, Chris Angelico wrote:
>
> > On Sun, Nov 27, 2016 at 11:13 AM, Steve D'Aprano
> > <steve+python at pearwood.info> wrote:
> >> So-called f-strings haven't even hit the  already been implicated in a
> >> code-injection vulnerability:
> >>
> >> http://bugs.python.org/issue28563
> >>
> >> I feel kind of vindicated here, because when so-called f-strings were
> >> first proposed I asked about the security implication of another way of
> >> evaluating arbitrary expressions, and I was told that there were no
> >> security implications. Technically that might be true in the sense that
> >> f-strings don't do anything that wasn't already possible, but as the
> >> above bug shows, they can make exploiting code injection trivially easy
> >> in cases where they were previously diabolically hard.
> >
> > Given that the exploit exists in 2.7, I would say f-strings didn't
> > create this, eval did.
>
> I never said that f-strings caused the vulnerability. I choose my words
> carefully. As I said when I mentioned this issue three weeks ago, the
> underlying cause of the vulnerability is the use of eval on an untrusted
> string. But the existence of a theoretical vulnerability is not the same as
> an exploit, let alone an easy exploit.
>
>
> > The problem is that you absolutely CANNOT
> > "sanitize" something before giving it to eval.
>
> Be careful about making absolute claims. I challenge you to break this use
> of eval:
>
> def calculate(phrase):
>     try:
>         phrase = sanitize(phrase)
>     except ValueError:
>         return
>     return eval(phrase, {'x': 20})
>
>
> def sanitize(phrase):
>     phrase = phrase.replace(' ', '')
>     if phrase in ('x+1', '2*x'):
>         return phrase
>     raise ValueError('unsafe phrase')
>
>
> For a more practical example, namedtuple uses exec to dynamically build the
> class. Can you find a code injection attack in namedtuple? I doubt it. Not
> all uses of exec or eval lead to a code injection vulnerability.
>
>
> > An f-string slips past the sanitizer, but so do other things.
>
> I daresay you are right that a sufficiently clever adversary may have found
> an exploit. But there's no sign that anyone actually did find an exploit,
> until f-strings made exploiting this trivial.
>
>
>
>
>
> --
> Steve
> ?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
> enough, things got worse.
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From torriem at gmail.com  Sat Nov 26 20:50:08 2016
From: torriem at gmail.com (Michael Torrie)
Date: Sat, 26 Nov 2016 18:50:08 -0700
Subject: The Case Against Python 3
In-Reply-To: <CAPjBxg-FEHwDzP7_+Bp82dKkY9cJ8aoMZWMMfVrWW0iHDbf0Vg@mail.gmail.com>
References: <75df94b8-8af4-4de4-93a8-dd4de5d186b7@googlegroups.com>
 <b44e089f-c7e4-4fd7-8c78-585b8b02269a@googlegroups.com>
 <CALwzidnCSVg62MR8Q_nA02Pindci9279WajKnS3HBw3Ow4BkCA@mail.gmail.com>
 <mailman.17.1480147352.2073.python-list@python.org>
 <583a2527$0$22140$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmoYSS-FLsF8e_Gs_eeb3AeF2jKgUe9ZLGwpSw5KQyay5A@mail.gmail.com>
 <mailman.26.1480206315.2073.python-list@python.org>
 <583a348e$0$1617$c3e8da3$5496439d@news.astraweb.com>
 <CAPjBxg-FEHwDzP7_+Bp82dKkY9cJ8aoMZWMMfVrWW0iHDbf0Vg@mail.gmail.com>
Message-ID: <0f94ece6-0b4a-cf08-4c48-80fb27b4a21c@gmail.com>

On 11/26/2016 06:26 PM, Nathan Ernst wrote:
> Sure, what if the input used a double quote instead of single, cursory
> glance looks like it might vulnerable.

Either a single quote or a double quote would not pass the sanitizer. Or
am I misunderstanding you?


From cs at zip.com.au  Sat Nov 26 21:24:26 2016
From: cs at zip.com.au (Cameron Simpson)
Date: Sun, 27 Nov 2016 13:24:26 +1100
Subject: how do i fix this invalid arguement error
In-Reply-To: <amij3chi5llhsngv84jpc8e3g04hdv8qq0@4ax.com>
References: <amij3chi5llhsngv84jpc8e3g04hdv8qq0@4ax.com>
Message-ID: <20161127022426.GA52067@cskk.homeip.net>

On 26Nov2016 12:55, Dennis Lee Bieber <wlfraed at ix.netcom.com> wrote:
>On Sat, 26 Nov 2016 08:12:46 -0800 (PST), junkone1 at gmail.com declaimed the
>following:
>>    with open('\\192.168.0.1\fe18cb0618cabd41\ninjatrader$EURUSDTestRun 2016-11-25-11-11.csv','r') as f:
[...]
>	Second... does Python open() accept web addresses? What is at
>192.168.0.1 that you don't have some more sensible path to it? 192.168.0.1
>is a private local network address. You are on a Windows machine so if the
>file is available on a local network node you can probably mount that node
>(Computer/Map Network Drive) and access the file as something like
>Z:/ninjatrader...

I thought //192.168.0.1/foo/... was a network share, getting "foo" from host 
192.168.0.1. Disclaimer: not a Windows guy. But I've certainly using UNIXy 
systems implementing this syntax, and thought I'd seen an equivalent in Windows 
land. So not HTTP, SMB. Confirmation or refutation from anyone?

Cheers,
Cameron Simpson <cs at zip.com.au>


From rosuav at gmail.com  Sat Nov 26 21:59:46 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 27 Nov 2016 13:59:46 +1100
Subject: how do i fix this invalid arguement error
In-Reply-To: <20161127022426.GA52067@cskk.homeip.net>
References: <amij3chi5llhsngv84jpc8e3g04hdv8qq0@4ax.com>
 <20161127022426.GA52067@cskk.homeip.net>
Message-ID: <CAPTjJmpq4wmLgwXM5aYnuMWL8nxEt_U0XaY5d1F5QE4U98qt9g@mail.gmail.com>

On Sun, Nov 27, 2016 at 1:24 PM, Cameron Simpson <cs at zip.com.au> wrote:
> I thought //192.168.0.1/foo/... was a network share, getting "foo" from host
> 192.168.0.1. Disclaimer: not a Windows guy. But I've certainly using UNIXy
> systems implementing this syntax, and thought I'd seen an equivalent in
> Windows land. So not HTTP, SMB. Confirmation or refutation from anyone?

Confirmed, that's the NETBIOS-over-TCP syntax, which Microsoft has
maintained into the newer Windows drive-sharing protocols.

ChrisA


From nathan.ernst at gmail.com  Sat Nov 26 22:29:07 2016
From: nathan.ernst at gmail.com (Nathan Ernst)
Date: Sat, 26 Nov 2016 21:29:07 -0600
Subject: The Case Against Python 3
In-Reply-To: <0f94ece6-0b4a-cf08-4c48-80fb27b4a21c@gmail.com>
References: <75df94b8-8af4-4de4-93a8-dd4de5d186b7@googlegroups.com>
 <b44e089f-c7e4-4fd7-8c78-585b8b02269a@googlegroups.com>
 <CALwzidnCSVg62MR8Q_nA02Pindci9279WajKnS3HBw3Ow4BkCA@mail.gmail.com>
 <mailman.17.1480147352.2073.python-list@python.org>
 <583a2527$0$22140$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmoYSS-FLsF8e_Gs_eeb3AeF2jKgUe9ZLGwpSw5KQyay5A@mail.gmail.com>
 <mailman.26.1480206315.2073.python-list@python.org>
 <583a348e$0$1617$c3e8da3$5496439d@news.astraweb.com>
 <CAPjBxg-FEHwDzP7_+Bp82dKkY9cJ8aoMZWMMfVrWW0iHDbf0Vg@mail.gmail.com>
 <0f94ece6-0b4a-cf08-4c48-80fb27b4a21c@gmail.com>
Message-ID: <CAPjBxg_fwdU5mbLaHjmdtmXhYyX2HxhDo3hnhD9U75SUcxCTfA@mail.gmail.com>

You're right. Didn't look closely enough at it in my phone. Still don't
think i'd recommend this in a general solution, though. You effectively
have to white-list code snippets. Not very useful.

On Nov 26, 2016 7:51 PM, "Michael Torrie" <torriem at gmail.com> wrote:

> On 11/26/2016 06:26 PM, Nathan Ernst wrote:
> > Sure, what if the input used a double quote instead of single, cursory
> > glance looks like it might vulnerable.
>
> Either a single quote or a double quote would not pass the sanitizer. Or
> am I misunderstanding you?
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From vek.m1234 at gmail.com  Sun Nov 27 04:53:06 2016
From: vek.m1234 at gmail.com (Veek M)
Date: Sun, 27 Nov 2016 15:23:06 +0530
Subject: dictionary mutability, hashability, __eq__, __hash__
Message-ID: <o1eabm$5bj$1@dont-email.me>

I was reading this: http://stackoverflow.com/questions/4418741/im-able-to-use-a-mutable-object-as-a-dictionary-key-in-python-is-this-not-disa

In a User Defined Type, one can provide __hash__ that returns a integer 
as a key to a dictionary.

so: d = { key : value }

What is the significance of __eq__ in this regard? Why does the article 
state:

"A class that overrides __eq__() and does not define __hash__() will 
have its __hash__() implicitly set to None. '

Is this behavior like __len__ and __bool__ wrt Boolean Expressions?
If one does: if x < y and __bool__ is not defined then __len__ is 
checked. Here, if __hash__ is undefined __eq__ is checked? What does he 
mean by 'overrides __eq__'??


Also didn't follow this 
" For classes you write, this method defaults to returning a value based 
off id(self), and *********** if equality is not determined by identity 
for those classes, you may be surprised by using them as keys: 
*******************"

"The requirement is that the hash of an object doesn't change over time, 
and ********* that it keeps comparing equal (==) with its original 
value. **************************"

Also if one can do x.a = 10 or 20 or whatever, and the class instance is 
mutable, then why do books keep stating that keys need to be immutable?
After all, __hash__ is the guy doing all the work and maintaining 
consistency for us. One could do:

class Fruit:
  editable_value = ''
def __hash__(self):
 if 'apple' in self.value: 
   return 10
 elif 'banana' in self.value:
   return 20


 and use 'apple' 'bannana' as keys for whatever mutable data..
Are the books wrong?



From jussi.piitulainen at helsinki.fi  Sun Nov 27 07:14:55 2016
From: jussi.piitulainen at helsinki.fi (Jussi Piitulainen)
Date: Sun, 27 Nov 2016 14:14:55 +0200
Subject: dictionary mutability, hashability, __eq__, __hash__
References: <o1eabm$5bj$1@dont-email.me>
Message-ID: <lf5h96t3yrk.fsf@ling.helsinki.fi>

Veek M writes:

[snip]

> Also if one can do x.a = 10 or 20 or whatever, and the class instance
> is mutable, then why do books keep stating that keys need to be
> immutable?  After all, __hash__ is the guy doing all the work and
> maintaining consistency for us. One could do:
>
> class Fruit:
>   editable_value = ''
> def __hash__(self):
>  if 'apple' in self.value: 
>    return 10
>  elif 'banana' in self.value:
>    return 20
>
>
>  and use 'apple' 'bannana' as keys for whatever mutable data..
> Are the books wrong?

The hash does not do all the work, and the underlying implementation of
a dictionary does not react appropriately to a key changing its hash
value. You could experiment further to see for yourself.

Here's a demonstration that Python's dictionary retains both keys after
they are mutated so that they become equal, yet finds neither key
(because they are not physically where their new hash value indicates).

I edited your class so that its methods manipulate an attribute that it
actually has, all hash values are integers, constructor takes an initial
value, objects are equal if their values are equal, and the written
representation of an object shows the value (I forgot quotes).

test = { Fruit('apple') : 'one', Fruit('orange') : 'two' }

print(test)
print(test[Fruit('orange')])
# prints:
# {Fruit(apple): 'one', Fruit(orange): 'two'}
# two

for key in test: key.value = 'banana'

print(test)
print(test[Fruit('banana')])

# prints:
# {Fruit(banana): 'one', Fruit(banana): 'two'}
# Traceback (most recent call last):
#   File "hash.py", line 25, in <module>
#     print(test[Fruit('banana')])
# KeyError: Fruit(banana)


From eryksun at gmail.com  Sun Nov 27 09:59:20 2016
From: eryksun at gmail.com (eryk sun)
Date: Sun, 27 Nov 2016 14:59:20 +0000
Subject: how do i fix this invalid arguement error
In-Reply-To: <amij3chi5llhsngv84jpc8e3g04hdv8qq0@4ax.com>
References: <2cb245b2-98c3-41e7-8c01-0ece703fc72d@googlegroups.com>
 <amij3chi5llhsngv84jpc8e3g04hdv8qq0@4ax.com>
Message-ID: <CACL+1au_=P7W6ygMxOJHd5CCX26u4YpvCvGZkUDMG10THfrB+Q@mail.gmail.com>

On Sat, Nov 26, 2016 at 5:55 PM, Dennis Lee Bieber
<wlfraed at ix.netcom.com> wrote:
> On Sat, 26 Nov 2016 08:12:46 -0800 (PST), junkone1 at gmail.com declaimed the
> following:
>
>> with open('\\192.168.0.1\fe18cb0618cabd41\ninjatrader$EURUSDTestRun
>> 2016-11-25-11-11.csv','r') as f:
>
> Second... does Python open() accept web addresses? What is at
> 192.168.0.1 that you don't have some more sensible path to it? 192.168.0.1
> is a private local network address. You are on a Windows machine so if the
> file is available on a local network node you can probably mount that node
> (Computer/Map Network Drive) and access the file as something like
> Z:/ninjatrader...

r"\\192.168.0.1\fe18..." is a UNC (Universal Naming Convention) path,
as defined by [MS-DTYP], Windows Data Types [1].

[1]: https://msdn.microsoft.com/en-us/library/gg465305

On Windows, open() calls CreateFile, which first translates DOS paths
to native paths that the kernel understands. UNC paths are converted
to the form r"\??\UNC\<Server>\<Share>\<Path>".

In kernel mode, the object manager resolves the virtual r"\??"
directory by checking for a "UNC" object in either the logon-session
DosDevices directory of the calling process/thread or the r"\GLOBAL??"
directory. r"\??\UNC" should almost always resolve to
r"\GLOBAL??\UNC". This is an object symbolic link to the native MUP
device (Multiple UNC Provider) [2], so the final path is
r"\Device\Mup\<Server>\<Share>\<Path>".

[2]: https://msdn.microsoft.com/en-us/library/ff556761

MUP resolves the <Server>\<Share> prefix to a registered provider
device, such as LanmanRedirector (SMB/CIFS), MailslotRedirector,
WebDavRedirector, RdpDr (RDP device redirector), or Csc (client-side
caching / offline files). If you're running Windows as a VirtualBox
guest, then your system may also have a VboxMiniRdr provider for
accessing folders shared by the host OS.

As to mount points, for many years Windows has provided flexible
alternatives to classic drive-letter (i.e. logical) mount points. I
prefer to mount UNC paths as symbolic links via `mklink /d` and local
volumes as junctions via `mountvol`.


From vek.m1234 at gmail.com  Sun Nov 27 10:28:57 2016
From: vek.m1234 at gmail.com (Veek M)
Date: Sun, 27 Nov 2016 20:58:57 +0530
Subject: dictionary mutability, hashability, __eq__, __hash__
References: <o1eabm$5bj$1@dont-email.me> <lf5h96t3yrk.fsf@ling.helsinki.fi>
Message-ID: <o1eu1g$a19$1@dont-email.me>

Jussi Piitulainen wrote:

> Veek M writes:
> 
> [snip]
> 
>> Also if one can do x.a = 10 or 20 or whatever, and the class instance
>> is mutable, then why do books keep stating that keys need to be
>> immutable?  After all, __hash__ is the guy doing all the work and
>> maintaining consistency for us. One could do:
>>
>> class Fruit:
>>   editable_value = ''
>> def __hash__(self):
>>  if 'apple' in self.value:
>>    return 10
>>  elif 'banana' in self.value:
>>    return 20
>>
>>
>>  and use 'apple' 'bannana' as keys for whatever mutable data..
>> Are the books wrong?
> 
> The hash does not do all the work, and the underlying implementation
> of a dictionary does not react appropriately to a key changing its
> hash value. You could experiment further to see for yourself.
> 
> Here's a demonstration that Python's dictionary retains both keys
> after they are mutated so that they become equal, yet finds neither
> key (because they are not physically where their new hash value
> indicates).
> 
> I edited your class so that its methods manipulate an attribute that
> it actually has, all hash values are integers, constructor takes an
> initial value, objects are equal if their values are equal, and the
> written representation of an object shows the value (I forgot quotes).
> 
> test = { Fruit('apple') : 'one', Fruit('orange') : 'two' }
> 
> print(test)
> print(test[Fruit('orange')])
> # prints:
> # {Fruit(apple): 'one', Fruit(orange): 'two'}
> # two
> 
> for key in test: key.value = 'banana'
> 
> print(test)
> print(test[Fruit('banana')])
> 
> # prints:
> # {Fruit(banana): 'one', Fruit(banana): 'two'}
> # Traceback (most recent call last):
> #   File "hash.py", line 25, in <module>
> #     print(test[Fruit('banana')])
> # KeyError: Fruit(banana)

ah! not so: that's because you are messing/changing the integer value 
for the key. If apple-object was returning 10, you can't then return 20 
(the text mangling seems to be completely irrelevant except you need it 
to figure out which integer to return but barring that..).

Here's an example of what you're doing (note 'fly' is returning 20 BUT 
the object-instance is 'apple' - that obviously won't work and has 
nothing to do with my Q, err.. (don't mean to be rude):
class Fruit(object):
    def __init__(self, text):
        self.text = text
        
    def mangle(self,text):
        self.text = text
        
    def __hash__(self):
        if 'apple' in self.text:
            return 10
        elif 'orange' in self.text:
            return 20
        elif 'fly' in self.text:
            return 20
        else:
            pass
        
apple = Fruit('apple')
orange = Fruit('orange')

d = { apple : 'APPLE_VALUE', orange : 'ORANGE_VALUE' }
print d

apple.mangle('fly')
print d[apple]

The Question is specific.. what I'm saying is that you can change 
attributes and the contents and totally mash the object up, so long as 
__hash__ returns the same integer for the same object. Correct?

Where does __eq__ fit in all this?




From vek.m1234 at gmail.com  Sun Nov 27 10:39:12 2016
From: vek.m1234 at gmail.com (Veek M)
Date: Sun, 27 Nov 2016 21:09:12 +0530
Subject: dictionary mutability, hashability, __eq__, __hash__
References: <o1eabm$5bj$1@dont-email.me> <lf5h96t3yrk.fsf@ling.helsinki.fi>
 <o1eu1g$a19$1@dont-email.me>
Message-ID: <o1eukn$dn9$1@dont-email.me>

Veek M wrote:

> Jussi Piitulainen wrote:
> 
>> Veek M writes:
>> 
>> [snip]
>> 
>>> Also if one can do x.a = 10 or 20 or whatever, and the class
>>> instance is mutable, then why do books keep stating that keys need
>>> to be
>>> immutable?  After all, __hash__ is the guy doing all the work and
>>> maintaining consistency for us. One could do:
>>>
>>> class Fruit:
>>>   editable_value = ''
>>> def __hash__(self):
>>>  if 'apple' in self.value:
>>>    return 10
>>>  elif 'banana' in self.value:
>>>    return 20
>>>
>>>
>>>  and use 'apple' 'bannana' as keys for whatever mutable data..
>>> Are the books wrong?
>> 
>> The hash does not do all the work, and the underlying implementation
>> of a dictionary does not react appropriately to a key changing its
>> hash value. You could experiment further to see for yourself.
>> 
>> Here's a demonstration that Python's dictionary retains both keys
>> after they are mutated so that they become equal, yet finds neither
>> key (because they are not physically where their new hash value
>> indicates).
>> 
>> I edited your class so that its methods manipulate an attribute that
>> it actually has, all hash values are integers, constructor takes an
>> initial value, objects are equal if their values are equal, and the
>> written representation of an object shows the value (I forgot
>> quotes).
>> 
>> test = { Fruit('apple') : 'one', Fruit('orange') : 'two' }
>> 
>> print(test)
>> print(test[Fruit('orange')])
>> # prints:
>> # {Fruit(apple): 'one', Fruit(orange): 'two'}
>> # two
>> 
>> for key in test: key.value = 'banana'
>> 
>> print(test)
>> print(test[Fruit('banana')])
>> 
>> # prints:
>> # {Fruit(banana): 'one', Fruit(banana): 'two'}
>> # Traceback (most recent call last):
>> #   File "hash.py", line 25, in <module>
>> #     print(test[Fruit('banana')])
>> # KeyError: Fruit(banana)
> 
> ah! not so: that's because you are messing/changing the integer value
> for the key. If apple-object was returning 10, you can't then return
> 20 (the text mangling seems to be completely irrelevant except you
> need it to figure out which integer to return but barring that..).
> 
> Here's an example of what you're doing (note 'fly' is returning 20 BUT
> the object-instance is 'apple' - that obviously won't work and has
> nothing to do with my Q, err.. (don't mean to be rude):
> class Fruit(object):
>     def __init__(self, text):
>         self.text = text
>         
>     def mangle(self,text):
>         self.text = text
>         
>     def __hash__(self):
>         if 'apple' in self.text:
>             return 10
>         elif 'orange' in self.text:
>             return 20
>         elif 'fly' in self.text:
>             return 20
>         else:
>             pass
>         
> apple = Fruit('apple')
> orange = Fruit('orange')
> 
> d = { apple : 'APPLE_VALUE', orange : 'ORANGE_VALUE' }
> print d
> 
> apple.mangle('fly')
> print d[apple]
> 
> The Question is specific.. what I'm saying is that you can change
> attributes and the contents and totally mash the object up, so long as
> __hash__ returns the same integer for the same object. Correct?
> 
> Where does __eq__ fit in all this?

How does MUTABILITY play a role in this?? __hash__ *HAS* to return an 
integer which is immutable anyhow.. And as far as UDT's are concerned 
they don't need to be immutable, they just need to return their integer-
key correctly..


From ned at nedbatchelder.com  Sun Nov 27 10:50:14 2016
From: ned at nedbatchelder.com (Ned Batchelder)
Date: Sun, 27 Nov 2016 07:50:14 -0800 (PST)
Subject: dictionary mutability, hashability, __eq__, __hash__
In-Reply-To: <o1eabm$5bj$1@dont-email.me>
References: <o1eabm$5bj$1@dont-email.me>
Message-ID: <742e63b0-95ce-4e88-a317-534d0b6e93ab@googlegroups.com>

On Sunday, November 27, 2016 at 4:53:20 AM UTC-5, Veek M wrote:
> I was reading this: http://stackoverflow.com/questions/4418741/im-able-to-use-a-mutable-object-as-a-dictionary-key-in-python-is-this-not-disa
> 
> In a User Defined Type, one can provide __hash__ that returns a integer 
> as a key to a dictionary.

This is not correct: the value returned by __hash__ is not the key.  The
object is the key, and the dictionary uses the hash value internally. It
also uses equality comparison.

There are two rules for using an object as a key in a dictionary:

1) The things an object is equal to must not change over the lifetime
of the object.

2) If two objects compare equal (with ==, that is to say, with __eq__),
then they must have equal __hash__ values.

Dictionaries consider two values the same key if they compare equal with
__eq__.  The __hash__ value is used as a critical optimization, which is
why these two rules must hold.

Rule 1 is hard to express succinctly, but it means that whatever
characteristics of the object are included in the equality
comparison, those characteristics must not change.  Immutable objects,
with no changeable characteristics at all, manage this easily.

--Ned.


From jussi.piitulainen at helsinki.fi  Sun Nov 27 11:41:14 2016
From: jussi.piitulainen at helsinki.fi (Jussi Piitulainen)
Date: Sun, 27 Nov 2016 18:41:14 +0200
Subject: dictionary mutability, hashability, __eq__, __hash__
References: <o1eabm$5bj$1@dont-email.me> <lf5h96t3yrk.fsf@ling.helsinki.fi>
 <o1eu1g$a19$1@dont-email.me>
Message-ID: <lf5vav898ph.fsf@ling.helsinki.fi>

Veek M writes:

> Jussi Piitulainen wrote:
>
>> Veek M writes:
>> 
>> [snip]
>> 
>>> Also if one can do x.a = 10 or 20 or whatever, and the class instance
>>> is mutable, then why do books keep stating that keys need to be
>>> immutable?  After all, __hash__ is the guy doing all the work and
>>> maintaining consistency for us. One could do:
>>>
>>> class Fruit:
>>>   editable_value = ''
>>> def __hash__(self):
>>>  if 'apple' in self.value:
>>>    return 10
>>>  elif 'banana' in self.value:
>>>    return 20
>>>
>>>
>>>  and use 'apple' 'bannana' as keys for whatever mutable data..
>>> Are the books wrong?
>> 
>> The hash does not do all the work, and the underlying implementation
>> of a dictionary does not react appropriately to a key changing its
>> hash value. You could experiment further to see for yourself.
>> 
>> Here's a demonstration that Python's dictionary retains both keys
>> after they are mutated so that they become equal, yet finds neither
>> key (because they are not physically where their new hash value
>> indicates).
>> 
>> I edited your class so that its methods manipulate an attribute that
>> it actually has, all hash values are integers, constructor takes an
>> initial value, objects are equal if their values are equal, and the
>> written representation of an object shows the value (I forgot quotes).
>> 
>> test = { Fruit('apple') : 'one', Fruit('orange') : 'two' }
>> 
>> print(test)
>> print(test[Fruit('orange')])
>> # prints:
>> # {Fruit(apple): 'one', Fruit(orange): 'two'}
>> # two
>> 
>> for key in test: key.value = 'banana'
>> 
>> print(test)
>> print(test[Fruit('banana')])
>> 
>> # prints:
>> # {Fruit(banana): 'one', Fruit(banana): 'two'}
>> # Traceback (most recent call last):
>> #   File "hash.py", line 25, in <module>
>> #     print(test[Fruit('banana')])
>> # KeyError: Fruit(banana)
>
> ah! not so: that's because you are messing/changing the integer value 
> for the key. If apple-object was returning 10, you can't then return 20 
> (the text mangling seems to be completely irrelevant except you need it 
> to figure out which integer to return but barring that..).

It was my best guess to what you intended __hash__ to be. You took that
risk when you posted obviously broken code.

Your new __hash__ function below behaves the same way.

> Here's an example of what you're doing (note 'fly' is returning 20 BUT 
> the object-instance is 'apple' - that obviously won't work and has 
> nothing to do with my Q, err.. (don't mean to be rude):
> class Fruit(object):
>     def __init__(self, text):
>         self.text = text
>         
>     def mangle(self,text):
>         self.text = text
>         
>     def __hash__(self):
>         if 'apple' in self.text:
>             return 10
>         elif 'orange' in self.text:
>             return 20
>         elif 'fly' in self.text:
>             return 20
>         else:
>             pass
>         
> apple = Fruit('apple')
> orange = Fruit('orange')
>
> d = { apple : 'APPLE_VALUE', orange : 'ORANGE_VALUE' }
> print d
>
> apple.mangle('fly')
> print d[apple]

Did you bother to try that? I get a KeyError (because the hash value of
the key object has changed).

> The Question is specific.. what I'm saying is that you can change
> attributes and the contents and totally mash the object up, so long as
> __hash__ returns the same integer for the same object. Correct?

Your __hash__ doesn't.

In your own example just above, you get 10 before mangling, and 20
after.

> Where does __eq__ fit in all this?

Make two different objects hash the same. Make them be __eq__ by
mangling them when they are already keys. See if you can still use both
as an index to get at the associated value. (You can't.)

But yes, you should be free to mutate fields that do not affect hashing
and equality. Object identity should work, if you are otherwise happy to
use object identity.


From al.basili at gmail.com  Sun Nov 27 16:44:32 2016
From: al.basili at gmail.com (alb)
Date: 27 Nov 2016 21:44:32 GMT
Subject: [sphinx] generating doc from neighboring folders
Message-ID: <ea12dtFquouU2@mid.individual.net>

Hi there,

we are evaluating the possibility to use Sphinx and rst to document our
projects. What we have is a project structure that would look like this:

./sandbox/project/
??? components
??? ??? module1
??? ??? ??? doc
??? ??? ??? ??? module1.rst
??? ??? ??? src
??? ??? module2
???     ??? doc
???     ??? ??? module2.rst
???     ??? src
??? doc
    ??? spec
        ??? conf.py
        ??? project_spec.rst


Since components are reusable amongs projects, we would like to get them
documented properly with their own conf.py, but then how would it be possible to
'include' them in the project documentation (under project/doc)?

It looks like sphinx wants all the sources under the same directory root.

What we thought of is writing a python script to traverse the project folder and
look for rst files and _copy_ them in the doc directory, but I find this
solution rather ugly.

Any ideas on how we should organize the rst files to accomplish that?

As a side note, I would like to include automatically links to my components
documentation in the index.rst file, any suggestion on how to do that?

Any suggestion/hint/comment is appreciated.

Al

-- 
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?


From gbpal13 at gmail.com  Mon Nov 28 00:52:50 2016
From: gbpal13 at gmail.com (g thakuri)
Date: Mon, 28 Nov 2016 11:22:50 +0530
Subject: correct way to catch exception with Python 'with' statement
Message-ID: <CALM1kKUJgM-FLyquTMDCpa=q2JVnfMBhAkgvH=rCX35QqSaK0Q@mail.gmail.com>

Dear Python friends,

Any suggestion on how to add exception and make the below program look
better  ,  I am using Python 2.7 and Linux

--------------------------------------------------------------------------------------------

def create_files_append():
    """  """
    try:
        os.makedirs(QA_TEST_DIR)
    except:
        raise OSError("Can't create directory (%s)!" % (QA_TEST_DIR))

    # Create few files and write something
    for i in range(1000):
        with open(os.path.join(QA_TEST_DIR,"filename%d" %i),'w') as f:
             f.write("hello")

    # Append the files
    for i in range(1000):
        with open(os.path.join(QA_TEST_DIR,"filename%d" %i),'w') as f:
             f.write("hello")

    return True

-----------------------------------------------------------------------------------------------

What will be the best  way to catch the exception in the above program  ?
Can we replace both the  with statement in the above program with something
like below

try:
    for i in range(1000):
               with open(os.path.join(QA_TEST_DIR,"filename%d" %i),'w') as f:
                   f.write("hello")
except IOError as e:
    raise

Regards,
PT


From ganesh1pal at gmail.com  Mon Nov 28 01:09:07 2016
From: ganesh1pal at gmail.com (Ganesh Pal)
Date: Mon, 28 Nov 2016 11:39:07 +0530
Subject: correct way to catch exception with Python 'with' statement
Message-ID: <CACT3xuX4J3maSiMgvzzSDG-xwJB2PKgyuUpMXbOO1-8mN+Bifg@mail.gmail.com>

Dear Python friends,

Any suggestion on how to add exception and make the below program look
better  ,  I am using Python 2.7 and Linux

------------------------------------------------------------
--------------------------------

def create_files_append():
    """  """
    try:
        os.makedirs(QA_TEST_DIR)
    except:
        raise OSError("Can't create directory (%s)!" % (QA_TEST_DIR))

    # Create few files and write something
    for i in range(1000):
        with open(os.path.join(QA_TEST_DIR,"filename%d" %i),'w') as f:
             f.write("hello")

    # Append the files
    for i in range(1000):
        with open(os.path.join(QA_TEST_DIR,"filename%d" %i),'w') as f:
             f.write("hello")

    return True

------------------------------------------------------------
-----------------------------------

What will be the best  way to catch the exception in the above program  ?
Can we replace both the  with statement in the above program with something
like below

try:
    for i in range(1000):
               with open(os.path.join(QA_TEST_DIR,"filename%d" %i),'w') as f:
                   f.write("hello")
except IOError as e:
    raise

Regards,
Ganesh


From steve+comp.lang.python at pearwood.info  Mon Nov 28 02:46:59 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Mon, 28 Nov 2016 18:46:59 +1100
Subject: correct way to catch exception with Python 'with' statement
References: <CACT3xuX4J3maSiMgvzzSDG-xwJB2PKgyuUpMXbOO1-8mN+Bifg@mail.gmail.com>
 <mailman.33.1480313687.2073.python-list@python.org>
Message-ID: <583be0f6$0$22141$c3e8da3$5496439d@news.astraweb.com>

On Monday 28 November 2016 17:09, Ganesh Pal wrote:

> Dear Python friends,
> 
> Any suggestion on how to add exception and make the below program look
> better  ,  I am using Python 2.7 and Linux
> 
> ------------------------------------------------------------
> --------------------------------
> 
> def create_files_append():
>     """  """
>     try:
>         os.makedirs(QA_TEST_DIR)
>     except:
>         raise OSError("Can't create directory (%s)!" % (QA_TEST_DIR))
> 
>     # Create few files and write something
>     for i in range(1000):
>         with open(os.path.join(QA_TEST_DIR,"filename%d" %i),'w') as f:
>              f.write("hello")
> 
>     # Append the files
>     for i in range(1000):
>         with open(os.path.join(QA_TEST_DIR,"filename%d" %i),'w') as f:
>              f.write("hello")
> 
>     return True


There is no need to return True. The function either succeeds, or it raises an 
exception, so there is no need to return any value at all.

There is no need to catch the exception if you're not going to do anything with 
it. os.makedirs already raises OSError if there is a problem creating the 
directories, so all you are doing is catching a sensible, useful error message 
that tells you WHY it failed with a useless, generic error message that tells 
you nothing.

py> os.makedirs('/root/xyz')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/root/xyz'


Permission denied is *much* more useful than "cannot create directory".

Your comment says "append the files", but you're not appending to the files, 
you are overwriting them. So your code is better written like this:


def create_files_append():
    """Docstring"""
    os.makedirs(QA_TEST_DIR)
    # Create few files and write something
    for i in range(1000):
        with open(os.path.join(QA_TEST_DIR,"filename%d" %i),'w') as f:
            f.write("hello")


If you really want to create the file, close the file, then append to it, you 
can do this:

def create_files_append():
    """Docstring"""
    os.makedirs(QA_TEST_DIR)
    # Create few files and write something
    for i in range(1000):
        with open(os.path.join(QA_TEST_DIR,"filename%d" %i),'w') as f:
            f.write("hello")
    # Append to the files
    for i in range(1000):
        with open(os.path.join(QA_TEST_DIR,"filename%d" %i),'a') as f:
            f.write("hello")


but that's a waste of time, this is better:

def create_files_append():
    """Docstring"""
    os.makedirs(QA_TEST_DIR)
    # Create few files and write something
    for i in range(1000):
        with open(os.path.join(QA_TEST_DIR,"filename%d" %i),'w') as f:
            f.write("hellohello")


If you want to skip any files that can't be opened:

def create_files_append():
    """Docstring"""
    os.makedirs(QA_TEST_DIR)
    # Create few files and write something
    for i in range(1000):
        try:
            with open(os.path.join(QA_TEST_DIR,"filename%d" %i),'w') as f:
                f.write("hellohello")
        except (OSError, IOError):
            pass



> What will be the best  way to catch the exception in the above program  ?

Not catching it at all.


> Can we replace both the  with statement in the above program with something
> like below
> 
> try:
>     for i in range(1000):
>                with open(os.path.join(QA_TEST_DIR,"filename%d" %i),'w') as f:
>                    f.write("hello")
> except IOError as e:
>     raise

What's the point of that? All you are doing is catching the exception and then 
immediately raising it again. That's the same as not catching it.




-- 
Steven
"Ever since I learned about confirmation bias, I've been seeing 
it everywhere." - Jon Ronson



From PointedEars at web.de  Mon Nov 28 07:30:34 2016
From: PointedEars at web.de (Thomas 'PointedEars' Lahn)
Date: Mon, 28 Nov 2016 13:30:34 +0100
Subject: correct way to catch exception with Python 'with' statement
References: <CACT3xuX4J3maSiMgvzzSDG-xwJB2PKgyuUpMXbOO1-8mN+Bifg@mail.gmail.com>
 <mailman.33.1480313687.2073.python-list@python.org>
Message-ID: <3959376.LvFx2qVVIh@PointedEars.de>

Ganesh Pal wrote:

> I am using Python 2.7 and Linux

As a rule of thumb?, use at least Python 3.3 for new programs.

> What will be the best  way to catch the exception in the above program  ?
> Can we replace both the  with statement in the above program with
> something like below
> 
> try:
>     for i in range(1000):
>                with open(os.path.join(QA_TEST_DIR,"filename%d" %i),'w') as
>                f:
>                    f.write("hello")
> except IOError as e:
>     raise

If you do it like this, you will have a hard time figuring out what caused 
the exception, and it becomes the harder the more you do in the loop. 
Because *several* statements could throw an exception of the *same* type.

Therefore, as a rule of thumb?, if you catch exceptions, catch them closest 
to where they could occur:

for i in range(1000):
    try:
        with open(os.path.join(QA_TEST_DIR, "filename%d" % i), 'w') as f:
            try:
                f.write("hello")
            except ? as e:
                raise e

    except ? as e:
        raise e

This is just an example; you do not have to just re-raise the exception; 
?raise? is just where your custom exception handling, if any, should be.  
For example, if f.write() fails with an IOError, you could decide to ignore 
that and ?continue? with the next file, whereas you may decide that failing 
to open a file is so grave a problem that you want to abort the program at 
this point.

Finally, use uniform indentation and spacing in your code; the former is 
even more important in Python than in other programming languages because 
indentation makes a block statement in Python.  Use a Python-aware editor to 
make it easy to create uniform indentation by use of the Tab key, and a 
reformatting feature for existing code.  I can recommend Atom [0] with the 
packages ?autocomplete-python?, ?linter-pylint?, ?python-isort?, and 
?python-tools? (?language-python? is a built-in); and Vim [1] with

if has("syntax")
    ?
    syntax on
    ?
endif
?
filetype plugin indent on

in ~/.vimrc and a ~/.vim/after/ftplugin/python.vim containing

setlocal expandtab
setlocal shiftwidth=4
setlocal softtabstop=4

_______
?  YMMV

[0] <https://atom.io/>
[1] <http://www.vim.org/>
-- 
PointedEars

Twitter: @PointedEars2
Please do not cc me. / Bitte keine Kopien per E-Mail.


From steve+python at pearwood.info  Mon Nov 28 07:48:30 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Mon, 28 Nov 2016 23:48:30 +1100
Subject: Asyncio -- delayed calculation
Message-ID: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>

I'm a complete and utter newbie when it comes to asynchronous programming,
so I may have the entire concept backwards here. But treat this as a
learning exercise rather than something I'd really do.

Suppose I have a bunch of calculations to do: count down from 10. So I have
a bunch of objects:

class Counter:
    def __init__(self):
        self.count = 10
    def count_down(self):
        print(self, "starting")
        while self.count > 0:
            # simulate a computation
            time.sleep(0.5)
            self.count -= 1
        print(self, "completed")

pool = [Counter() for i in range(5)]

for obj in pool:
    obj.count_down()



Since that's all blocking, I wait for the first Counter to count down to
zero before I move on to the second.

Let's pretend that the computation can be performed asynchronously, so that
I can have all five Counter objects counting down in parallel. I have this:


import asyncio

class Counter:
    def __init__(self):
        self.count = 10
    async def count_down(self):
        print(self, "starting")
        while self.count > 0:
            # simulate a computation
            await asyncio.sleep(0.5)
            self.count -= 1
        print(self, "completed")

async def main():
    pool = [Counter() for i in range(5)]
    for obj in pool:
        obj.count_down()

loop = asyncio.get_event_loop()
loop.run_until_complete(main())




When I try running that, I get no output. No error, no exception, the
run_until_complete simply returns instantly.

What am I doing wrong?

What I expected is that the pool of Counter objects would be created, each
one would have their count_down() method called without blocking, so I'd
have something like:

# IDs are simulated for ease of comprehension
<__main__.Counter object at 0x0123> starting
<__main__.Counter object at 0x0246> starting
<__main__.Counter object at 0x048c> starting
<__main__.Counter object at 0x0918> starting
<__main__.Counter object at 0x1230> starting
<__main__.Counter object at 0x0123> completed
<__main__.Counter object at 0x0246> completed
<__main__.Counter object at 0x048c> completed
<__main__.Counter object at 0x0918> completed
<__main__.Counter object at 0x1230> completed



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From rosuav at gmail.com  Mon Nov 28 08:03:58 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 29 Nov 2016 00:03:58 +1100
Subject: Asyncio -- delayed calculation
In-Reply-To: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>

On Mon, Nov 28, 2016 at 11:48 PM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> When I try running that, I get no output. No error, no exception, the
> run_until_complete simply returns instantly.

When I do, I get this warning:

asynctest.py:17: RuntimeWarning: coroutine 'Counter.count_down' was
never awaited
  obj.count_down()

Putting an 'await' in front of that call causes the tasks to be run
consecutively, of course. The most similar code for running tasks
concurrently seems to be this:

async def main():
    pool = [Counter() for i in range(5)]
    await asyncio.gather(*(obj.count_down() for obj in pool))

Taken from:
https://docs.python.org/3/library/asyncio-task.html#example-parallel-execution-of-tasks

There may be other ways, but that's the best I could find. It seems to
do what you want.

ChrisA


From ganesh1pal at gmail.com  Mon Nov 28 10:08:41 2016
From: ganesh1pal at gmail.com (Ganesh Pal)
Date: Mon, 28 Nov 2016 20:38:41 +0530
Subject: Simple Python equivalent for the shell command
Message-ID: <CACT3xuUqR3QfELaqgk73xd94-Rn3-mv35Y2cGAn=rveCi3E9Yw@mail.gmail.com>

I  was trying  to write a function that will return me the unique number
associated with each employee id.The command has the output in the below
pattern

Linux-Box-1# employee_details ls
List of names:
100910bd9 s7018
100d60003 s7019
110610bd3 s7020
100d60002 s7021


Linux-Box-1# employee_details ls | grep "s7020" | awk '{print $1}'
100d60003

It's a one liner in Shell  :)


I tried converting  the same  in the python style , Any better suggestion
and loop holes in the below program

def get_unique_number(str(emp_id)):
    """ Return the unique number associated with each employee id """
    out, err, rc = run("employee_details ls", timeout=600)
    emp_unum=""
    if rc != 0:
        return False

    for line in out.split('\n'):
        if emp_id in line:
           emp_unum = line.split()[0]
    return emp_unum

I am on Python 2.7 and Linux OS

Regards,
Ganesh


From ganesh1pal at gmail.com  Mon Nov 28 10:18:01 2016
From: ganesh1pal at gmail.com (Ganesh Pal)
Date: Mon, 28 Nov 2016 20:48:01 +0530
Subject: correct way to catch exception with Python 'with' statement
In-Reply-To: <583be0f6$0$22141$c3e8da3$5496439d@news.astraweb.com>
References: <CACT3xuX4J3maSiMgvzzSDG-xwJB2PKgyuUpMXbOO1-8mN+Bifg@mail.gmail.com>
 <mailman.33.1480313687.2073.python-list@python.org>
 <583be0f6$0$22141$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CACT3xuUKOEV44SEQ+8_VZKBNUG5d96SB0ztULKK=h1OdrGg5vw@mail.gmail.com>

On Mon, Nov 28, 2016 at 1:16 PM, Steven D'Aprano <
steve+comp.lang.python at pearwood.info> wrote:

>
>
> There is no need to return True. The function either succeeds, or it
> raises an
> exception, so there is no need to return any value at all.
>
>
 I returned True here ,because based on the result of this function , I
would want to perform next steps

 Example
                  if  create_files_append():
                       do_somthing()
                  else:
                         do_next_thing()



>
> Your comment says "append the files", but you're not appending to the
> files,
> you are overwriting them. So your code is better written like this:
>
>
 Yes , correct  and apologies there was a typo it should have been 'a'
instead of 'w'  . Thanks for the comments


From ganesh1pal at gmail.com  Mon Nov 28 10:26:08 2016
From: ganesh1pal at gmail.com (Ganesh Pal)
Date: Mon, 28 Nov 2016 20:56:08 +0530
Subject: Simple Python equivalent for the shell command
In-Reply-To: <CACT3xuUqR3QfELaqgk73xd94-Rn3-mv35Y2cGAn=rveCi3E9Yw@mail.gmail.com>
References: <CACT3xuUqR3QfELaqgk73xd94-Rn3-mv35Y2cGAn=rveCi3E9Yw@mail.gmail.com>
Message-ID: <CACT3xuVHiOXLbCxiVpJOnaAo7jbmiL7UrfUUxTN2zNUX=F_O1g@mail.gmail.com>

I  remembered that I might need to add an else condition if the emp_num
does not exist , so re sending the updated code


def get_unique_number(str(emp_id)):
    """ Return the unique number associated with each employee id """
    out, err, rc = run("employee_details ls", timeout=600)
    emp_unum=""
    if rc != 0:
        return False

    for line in out.split('\n'):
        if emp_id in line:
           emp_unum = line.split()[0]
        else:
                print("emp_unum does not exist")
                return False
    return emp_unum

PS :  [Edited the above code with else condition]

Regards,
Ganesh


On Mon, Nov 28, 2016 at 8:38 PM, Ganesh Pal <ganesh1pal at gmail.com> wrote:

>
>
> I  was trying  to write a function that will return me the unique number
> associated with each employee id.The command has the output in the below
> pattern
>
> Linux-Box-1# employee_details ls
> List of names:
> 100910bd9 s7018
> 100d60003 s7019
> 110610bd3 s7020
> 100d60002 s7021
>
>
> Linux-Box-1# employee_details ls | grep "s7020" | awk '{print $1}'
> 100d60003
>
> It's a one liner in Shell  :)
>
>
> I tried converting  the same  in the python style , Any better suggestion
> and loop holes in the below program
>
> def get_unique_number(str(emp_id)):
>     """ Return the unique number associated with each employee id """
>     out, err, rc = run("employee_details ls", timeout=600)
>     emp_unum=""
>     if rc != 0:
>         return False
>
>     for line in out.split('\n'):
>         if emp_id in line:
>            emp_unum = line.split()[0]
>     return emp_unum
>
> I am on Python 2.7 and Linux OS
>
> Regards,
> Ganesh
>


From ian.g.kelly at gmail.com  Mon Nov 28 10:53:00 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Mon, 28 Nov 2016 08:53:00 -0700
Subject: Asyncio -- delayed calculation
In-Reply-To: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CALwzidnTKni6SGB8Gpb0-zMdEkeqa1m6NN-ATfPkeY2YuNb1fg@mail.gmail.com>

On Mon, Nov 28, 2016 at 5:48 AM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> Let's pretend that the computation can be performed asynchronously, so that
> I can have all five Counter objects counting down in parallel. I have this:
>
>
> import asyncio
>
> class Counter:
>     def __init__(self):
>         self.count = 10
>     async def count_down(self):
>         print(self, "starting")
>         while self.count > 0:
>             # simulate a computation
>             await asyncio.sleep(0.5)
>             self.count -= 1
>         print(self, "completed")
>
> async def main():
>     pool = [Counter() for i in range(5)]
>     for obj in pool:
>         obj.count_down()
>
> loop = asyncio.get_event_loop()
> loop.run_until_complete(main())
>
>
>
>
> When I try running that, I get no output. No error, no exception, the
> run_until_complete simply returns instantly.
>
> What am I doing wrong?

Remember that coroutines are basically generators. Native "async def"
coroutines are dressed up as something different, but they were still
designed as a drop-in replacement for generator coroutines. If
count_down were a generator function then simply calling it wouldn't
really do anything and as a native coroutine it still doesn't (other
than return a "coroutine object").

In order for the coroutines to actually do anything, you need to
schedule them in some way with the event loop. That could take the
form of awaiting them from some other coroutine, or passing them
directly to loop.run_until_complete or event_loop.create_task, or as
Chris suggested awaiting them as an aggregate.


From torriem at gmail.com  Mon Nov 28 10:55:33 2016
From: torriem at gmail.com (Michael Torrie)
Date: Mon, 28 Nov 2016 08:55:33 -0700
Subject: Simple Python equivalent for the shell command
In-Reply-To: <CACT3xuUqR3QfELaqgk73xd94-Rn3-mv35Y2cGAn=rveCi3E9Yw@mail.gmail.com>
References: <CACT3xuUqR3QfELaqgk73xd94-Rn3-mv35Y2cGAn=rveCi3E9Yw@mail.gmail.com>
Message-ID: <d583b6da-2c6e-6b43-ec94-98791caab3f0@gmail.com>

On 11/28/2016 08:08 AM, Ganesh Pal wrote:
> I  was trying  to write a function that will return me the unique number
> associated with each employee id.The command has the output in the below
> pattern
> 
> Linux-Box-1# employee_details ls
> List of names:
> 100910bd9 s7018
> 100d60003 s7019
> 110610bd3 s7020
> 100d60002 s7021
> 
> 
> Linux-Box-1# employee_details ls | grep "s7020" | awk '{print $1}'
> 100d60003
> 
> It's a one liner in Shell  :)
> 
> 
> I tried converting  the same  in the python style , Any better suggestion
> and loop holes in the below program

Well Bash is really good at some things.  Piping commands together is
one of those things.  Python can do such things but not in as compact a
way.  For one Python has no quick way of interfacing with subprograms as
if they were language-level constructs like Bash does.  But it still can
do a lot of the same tasks that shell scripting does.  Read on.

In many respects, generators are the Python equivalent of pipes. See
this: http://www.dabeaz.com/generators/, specifically
http://www.dabeaz.com/generators/Generators.pdf

Here's a simple grep-like filter that searches lines:

def simple_grep(search_in, look_for):
    for line in search_in:
        if look_for in line:
            yield line

import sys
for result in simple_grep(sys.stdin, "s2070"):
    print result.split()[0]

This obviously does not run a subprogram, but could feed the filter
input from any file-like object that returns lines.  The beauty of
generators is that you can string them together just like you would do
in Bash with pipes.  And they can potentially be fairly fast, especially
if you can reduce the number of lines the generators have to process by
putting the faster filters earlier on in the chain (much like you would
optimize a database query by pushing selections ahead of joins).  Though
this is not shorter than your code, nor is it substantially different,
it does have the advantage that the simple_grep() generator can be used
in multiple places and could even be placed in utility library.



From torriem at gmail.com  Mon Nov 28 10:59:11 2016
From: torriem at gmail.com (Michael Torrie)
Date: Mon, 28 Nov 2016 08:59:11 -0700
Subject: correct way to catch exception with Python 'with' statement
In-Reply-To: <CACT3xuUKOEV44SEQ+8_VZKBNUG5d96SB0ztULKK=h1OdrGg5vw@mail.gmail.com>
References: <CACT3xuX4J3maSiMgvzzSDG-xwJB2PKgyuUpMXbOO1-8mN+Bifg@mail.gmail.com>
 <mailman.33.1480313687.2073.python-list@python.org>
 <583be0f6$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <CACT3xuUKOEV44SEQ+8_VZKBNUG5d96SB0ztULKK=h1OdrGg5vw@mail.gmail.com>
Message-ID: <e9e6d6c3-8c6c-80ff-0026-90550306d8a4@gmail.com>

On 11/28/2016 08:18 AM, Ganesh Pal wrote:
> On Mon, Nov 28, 2016 at 1:16 PM, Steven D'Aprano <
> steve+comp.lang.python at pearwood.info> wrote:
> 
>>
>>
>> There is no need to return True. The function either succeeds, or it
>> raises an
>> exception, so there is no need to return any value at all.
>>
>>
>  I returned True here ,because based on the result of this function , I
> would want to perform next steps

Except that you didn't read what Steven wrote, and you may not be
understanding your own code.  If the function fails it never returns
anyway, since an exception is raised!  So you don't need to worry about
a return type at all. You just do:

create_files_append()
do_something().

If you want to check for the exception, you would do:
try:
	create_files_append()
	do_something()
except OSError:
	do_next_thing()




From random832 at fastmail.com  Mon Nov 28 11:01:54 2016
From: random832 at fastmail.com (Random832)
Date: Mon, 28 Nov 2016 11:01:54 -0500
Subject: Immutability of Floats, Ints and Strings in Python
In-Reply-To: <4630ad9e-e336-4f78-aebb-706d403737ee@googlegroups.com>
References: <31b70ea6-d3ed-4756-81de-e6f327ebb56f@googlegroups.com>
 <4630ad9e-e336-4f78-aebb-706d403737ee@googlegroups.com>
Message-ID: <1480348914.3027650.801285233.1ED5FCA2@webmail.messagingengine.com>

On Fri, Nov 25, 2016, at 06:33, Ned Batchelder wrote:
> A Python implementation can choose when to reuse immutable objects and
> when not to.  Reusing a value has a cost, because the values have to
> be kept, and then found again. So the cost is only paid when there's
> a reasonable chance that the values will actually be needed again.
> And that cost has to be weighed against the opposite cost of simply
> making a new object instead.

Of course, there are more complicated costs to consider. For an
implementation where objects do not have a naturally persistent object
identity (such as an immovable address in memory as in cpython) they may
consider it easier to have an "object identity" that consists of the
whole value for immutable types rather than pay whatever costs are
associated with having a unique and unchanging "id" value. It's also not
hard to imagine an implementation that has "references" consisting of a
tagged union and incorporating the value in the "reference" itself (and
therefore the id) for floats and small integer/string values, though I
can't name any implementation (of Python, anyway - it's not uncommon for
Lisp) that does so.


From torriem at gmail.com  Mon Nov 28 11:17:50 2016
From: torriem at gmail.com (Michael Torrie)
Date: Mon, 28 Nov 2016 09:17:50 -0700
Subject: Simple Python equivalent for the shell command
In-Reply-To: <d583b6da-2c6e-6b43-ec94-98791caab3f0@gmail.com>
References: <CACT3xuUqR3QfELaqgk73xd94-Rn3-mv35Y2cGAn=rveCi3E9Yw@mail.gmail.com>
 <d583b6da-2c6e-6b43-ec94-98791caab3f0@gmail.com>
Message-ID: <c973acc2-1250-effc-7d66-f90e98c38516@gmail.com>

On 11/28/2016 08:55 AM, Michael Torrie wrote:
> Well Bash is really good at some things.  Piping commands together is
> one of those things.  Python can do such things but not in as compact a
> way.  For one Python has no quick way of interfacing with subprograms as
> if they were language-level constructs like Bash does.  But it still can
> do a lot of the same tasks that shell scripting does.  Read on.
> 
> In many respects, generators are the Python equivalent of pipes. See
> this: http://www.dabeaz.com/generators/, specifically
> http://www.dabeaz.com/generators/Generators.pdf

I should mention that the one thing that Python has no support for in
its generator methodology is the idea of having one execution unit that
can have both standard out and standard error pipes and communicate with
both at the same time. Bash can support arbitrary pipes I believe.
However even Bash's syntax is awkward, and I can't imagine a decent way
to do it in Python anyway.  Perhaps yielding a tuple.


From rachid.abkar at gmail.com  Mon Nov 28 12:04:30 2016
From: rachid.abkar at gmail.com (badr)
Date: Mon, 28 Nov 2016 09:04:30 -0800 (PST)
Subject: plot band structure
Message-ID: <614c8490-0d7c-44de-96f2-a656d66d570e@googlegroups.com>

Hello everybody
I have an xml file for Band structure that I would like to plot like this https://pypi.python.org/pypi/pydass_vasp/0.1. 
I have download phyton and I want to know the code to use and how to run the program, I am beginer to use it , please I need  a help,thank you in advance.

Regards

Rachid


From amirouche.boubekki at gmail.com  Mon Nov 28 14:02:22 2016
From: amirouche.boubekki at gmail.com (Amirouche Boubekki)
Date: Mon, 28 Nov 2016 19:02:22 +0000
Subject: ANN: JavaScrypthon 0.5, now with embedded evaluation of
 transpiled code
In-Reply-To: <87a8cmfkpg.fsf@metapensiero.it>
References: <87a8cmfkpg.fsf@metapensiero.it>
Message-ID: <CAL7_Mo-kMJqw8GK78nxcspUkvQ4aTGO5XC46Y6svki4mo5rPEQ@mail.gmail.com>

On Sat, Nov 26, 2016 at 7:21 PM Alberto Berti <
azazel+python-announce at arstecnica.it> wrote:

> Hi all,
>

H?llo!


> i'm pleased to announce that JavaScripthon 0.5 has been released to
> PyPI. JavaScrypthon can translate a subset of Python 3.5 code to  ES6
> JavaScript producing beautiful and lean  code, while supporting some of
> the latest Python features.
>
> Changelog
> (https://github.com/azazel75/metapensiero.pj/blob/master/CHANGES.rst)


This is the best of ES6 in a Python syntax.

Impressive work.

I have a few questions:

a) Why do you choose not to use your own Javascripton class to represent
Python dict?

b) Do you do some type tracing or something to be able to convert
``foo.update(bar)`` to ``Object.assign(foo, bar)``

c) IIUC you don't implement Python type system and rely on ES6 (or ES7)
class system? Is that correct? If that's the case how do you implement
class decorators and method decorators?

d) What about Python <-> Javascript interop. It's a tremendous topic not to
be forgotten.

A few remarks:

i) I had a remark about arguments but javascripthon doesn't support
**kwargs at call site. This is implemented in Pythonium, have a look, it
has some js <-> python interop.

ii) Too bad I can't run the vanilla pystone benchmark

I created a similar project targeting ES5 [0].

Also, FWIW users are looking for a Javascript replacement that is real
Python, not another coffeescript.

Best wishes!

[0] https://github.com/amirouche/pythonium/blob/master/dodge.py


From esj at harvee.org  Mon Nov 28 14:53:57 2016
From: esj at harvee.org (Eric S. Johansson)
Date: Mon, 28 Nov 2016 14:53:57 -0500
Subject: ANN: JavaScrypthon 0.5, now with embedded evaluation of
 transpiled code
In-Reply-To: <CAL7_Mo-kMJqw8GK78nxcspUkvQ4aTGO5XC46Y6svki4mo5rPEQ@mail.gmail.com>
References: <87a8cmfkpg.fsf@metapensiero.it>
 <CAL7_Mo-kMJqw8GK78nxcspUkvQ4aTGO5XC46Y6svki4mo5rPEQ@mail.gmail.com>
Message-ID: <22d051d3-1362-4a3b-b98d-13934c3a07f3@harvee.org>



On 11/28/2016 2:02 PM, Amirouche Boubekki wrote:
> Also, FWIW users are looking for a Javascript replacement that is real
> Python, not another coffeescript.
does this count? http://brython.info/



From greg.ewing at canterbury.ac.nz  Mon Nov 28 17:35:47 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Tue, 29 Nov 2016 11:35:47 +1300
Subject: The Case Against Python 3
In-Reply-To: <583a348e$0$1617$c3e8da3$5496439d@news.astraweb.com>
References: <75df94b8-8af4-4de4-93a8-dd4de5d186b7@googlegroups.com>
 <b44e089f-c7e4-4fd7-8c78-585b8b02269a@googlegroups.com>
 <CALwzidnCSVg62MR8Q_nA02Pindci9279WajKnS3HBw3Ow4BkCA@mail.gmail.com>
 <mailman.17.1480147352.2073.python-list@python.org>
 <583a2527$0$22140$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmoYSS-FLsF8e_Gs_eeb3AeF2jKgUe9ZLGwpSw5KQyay5A@mail.gmail.com>
 <mailman.26.1480206315.2073.python-list@python.org>
 <583a348e$0$1617$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <ea3pq5Fh9nuU1@mid.individual.net>

Steve D'Aprano wrote:
> I daresay you are right that a sufficiently clever adversary may have found
> an exploit. But there's no sign that anyone actually did find an exploit,
> until f-strings made exploiting this trivial.

The person who wrote the bug report found at least one
way of exploiting it that doesn't require f-strings.

I agree that f-strings are not to blame here. If we really
want to avoid breaking anyone's ill-conceived attempts at
sandboxing eval, we'd better not add anything more to the
language, ever, because nobody can foresee all the possible
consequences.

-- 
Greg


From no.email at nospam.invalid  Mon Nov 28 18:00:07 2016
From: no.email at nospam.invalid (Paul Rubin)
Date: Mon, 28 Nov 2016 15:00:07 -0800
Subject: The Case Against Python 3
References: <75df94b8-8af4-4de4-93a8-dd4de5d186b7@googlegroups.com>
 <b44e089f-c7e4-4fd7-8c78-585b8b02269a@googlegroups.com>
 <CALwzidnCSVg62MR8Q_nA02Pindci9279WajKnS3HBw3Ow4BkCA@mail.gmail.com>
 <mailman.17.1480147352.2073.python-list@python.org>
 <583a2527$0$22140$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmoYSS-FLsF8e_Gs_eeb3AeF2jKgUe9ZLGwpSw5KQyay5A@mail.gmail.com>
 <mailman.26.1480206315.2073.python-list@python.org>
 <583a348e$0$1617$c3e8da3$5496439d@news.astraweb.com>
 <ea3pq5Fh9nuU1@mid.individual.net>
Message-ID: <87a8cj8b2g.fsf@nightsong.com>

Gregory Ewing <greg.ewing at canterbury.ac.nz> writes:
> I agree that f-strings are not to blame here. If we really want to
> avoid breaking anyone's ill-conceived attempts at sandboxing eval,
> we'd better not add anything more to the language, ever, because
> nobody can foresee all the possible consequences.

I'm surprised eval was used like that.  It seems ill-advised.  Something
similar happened with pickles some time back.  Oh my, now I'm reminded
at how old we've all gotten:

    "Using eval this way is like storing a vat of cyanide in your
    child's bedroom.  Sure, maybe if you check the seals and locks on
    the vat carefully enough, you can convince yourself that your child
    won't be able to get to the cyanide.  But wouldn't you feel safer
    just not having the vat there at all?  That's basic
    safety-consciousness.  Security consciousness works the same way.
    Try to keep dangerous ingredients and attackers as far away from
    each other as possible."  ( http://bugs.python.org/msg6972 )


From steve+python at pearwood.info  Mon Nov 28 18:54:07 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Tue, 29 Nov 2016 10:54:07 +1100
Subject: The Case Against Python 3
References: <75df94b8-8af4-4de4-93a8-dd4de5d186b7@googlegroups.com>
 <b44e089f-c7e4-4fd7-8c78-585b8b02269a@googlegroups.com>
 <CALwzidnCSVg62MR8Q_nA02Pindci9279WajKnS3HBw3Ow4BkCA@mail.gmail.com>
 <mailman.17.1480147352.2073.python-list@python.org>
 <583a2527$0$22140$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmoYSS-FLsF8e_Gs_eeb3AeF2jKgUe9ZLGwpSw5KQyay5A@mail.gmail.com>
 <mailman.26.1480206315.2073.python-list@python.org>
 <583a348e$0$1617$c3e8da3$5496439d@news.astraweb.com>
 <ea3pq5Fh9nuU1@mid.individual.net>
Message-ID: <583cc3a1$0$1591$c3e8da3$5496439d@news.astraweb.com>

On Tue, 29 Nov 2016 09:35 am, Gregory Ewing wrote:

> Steve D'Aprano wrote:
>> I daresay you are right that a sufficiently clever adversary may have
>> found an exploit. But there's no sign that anyone actually did find an
>> exploit, until f-strings made exploiting this trivial.
> 
> The person who wrote the bug report found at least one
> way of exploiting it that doesn't require f-strings.

Oops, of course you are right. The bug report says:

    My first discovery was that nothing prevents an input plural 
    string that resembles a function call:

       gettext.c2py("n()")(lambda: os.system("sh"))

    This is of course a low risk bug, since it requires control
    of both the plural function string and the argument.

http://bugs.python.org/issue28563


And later on:

    Instead of passing a string to eval, we can build a string
    from characters in the docstrings available in the context
    of the gettext module:
    [...]
    This will successfully spawn a shell in Python 2.7.11.

Which I had forgotten about. (And by the way: the exploit could have been
avoided by refusing any string which contains an underscore. If you can't
avoid vulnerable code, at least make attackers work hard to exploit it.)

The point I was making was this comment:

    Bonus: With the new string interpolation in Python 3.7, 
    exploiting gettext.c2py becomes trivial:

       gettext.c2py('f"{os.system(\'sh\')}"')(0)

    The tokenizer will recognize the entire format-string as 
    JUST A STRING [emphasis added]

f-strings are *not* "just a string", as this shows -- they're actually
equivalent to a call to eval. And *that* is my point: we've picked a syntax
which looks like a static literal string for something which is not only a
function call but equivalent to the second most powerful (and hence
dangerous) built-in function call available in the language.

Normally I think Guido's instinct for syntax and language features is pretty
good, but here I think he's made a blunder. The original proposal on the
Python-Ideas mailing list was for syntax to automatically perform *only*
name lookups so that 

    f"{x}"

was syntactic sugar for

    "{x}".format(x=x)


and from that simple request it has grown in scope to the point that we can
now write:

    f"{os.system('sh')}"

as syntactic sugar for:

    str(eval("os.system('sh')"))

Yay for progress!



> I agree that f-strings are not to blame here.

I've already said that. Certainly the *vulnerability* comes from the use of
eval by gettext.c2py, but the TRIVIAL *exploit* comes from f-strings.


> If we really 
> want to avoid breaking anyone's ill-conceived attempts at
> sandboxing eval, we'd better not add anything more to the
> language, ever, because nobody can foresee all the possible
> consequences.

Now you're just being silly, this isn't "anything", it is a specific design
decision: something which looks like, and is treated by the tokeniser, as a
string but is actually a hidden call to eval.

And in fact I did foresee the consequences. I never predicted this *exact*
vulnerability, of course, but I did say that disguising a call to eval
inside something which looks like a string would lead to trouble. I was
poo-pooed for that idea, so please excuse me if I gloat a little when the
first "trouble" is discovered before the feature is even available in a
production release.

I don't say this as if it were an amazing feat of prediction. I'm astonished
that apparently others can't, or won't, see it. Its like when people store
weed killer or some other poison in a soft-drink bottle, complete with the
label still on, and then are surprised when someone gets poisoned.

http://www.dailymail.co.uk/news/article-1337101/Father-Phillip-Ward-dies-accidentally-drinking-weedkiller-Lucozade-bottle.html

(Not the first, or last, case of accidental poisoning from herbicide or
pesticide stored inappropriately.)



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Mon Nov 28 19:20:56 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Tue, 29 Nov 2016 11:20:56 +1100
Subject: Asyncio -- delayed calculation
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CALwzidnTKni6SGB8Gpb0-zMdEkeqa1m6NN-ATfPkeY2YuNb1fg@mail.gmail.com>
 <mailman.38.1480348423.2073.python-list@python.org>
Message-ID: <583cc9ea$0$22140$c3e8da3$5496439d@news.astraweb.com>

On Tue, 29 Nov 2016 02:53 am, Ian Kelly wrote:

> In order for the coroutines to actually do anything, you need to
> schedule them in some way with the event loop. That could take the
> form of awaiting them from some other coroutine, or passing them
> directly to loop.run_until_complete or event_loop.create_task, or as
> Chris suggested awaiting them as an aggregate.

I thought that's what I had done, by calling

loop.run_until_complete(main())



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From rosuav at gmail.com  Mon Nov 28 20:00:43 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 29 Nov 2016 12:00:43 +1100
Subject: The Case Against Python 3
In-Reply-To: <583cc3a1$0$1591$c3e8da3$5496439d@news.astraweb.com>
References: <75df94b8-8af4-4de4-93a8-dd4de5d186b7@googlegroups.com>
 <b44e089f-c7e4-4fd7-8c78-585b8b02269a@googlegroups.com>
 <CALwzidnCSVg62MR8Q_nA02Pindci9279WajKnS3HBw3Ow4BkCA@mail.gmail.com>
 <mailman.17.1480147352.2073.python-list@python.org>
 <583a2527$0$22140$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmoYSS-FLsF8e_Gs_eeb3AeF2jKgUe9ZLGwpSw5KQyay5A@mail.gmail.com>
 <mailman.26.1480206315.2073.python-list@python.org>
 <583a348e$0$1617$c3e8da3$5496439d@news.astraweb.com>
 <ea3pq5Fh9nuU1@mid.individual.net>
 <583cc3a1$0$1591$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPTjJmpuJqfvfb4=E3AA3OL4pdbH2HGP7qkTMgibYxWT=iNwzw@mail.gmail.com>

On Tue, Nov 29, 2016 at 10:54 AM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> Now you're just being silly, this isn't "anything", it is a specific design
> decision: something which looks like, and is treated by the tokeniser, as a
> string but is actually a hidden call to eval.
>

This, I think, is the crux. A "hidden eval" is a fundamentally bad
thing. Python 2's input() function is bad for this reason - not
because eval is necessarily evil (there are times when that's the
exact behaviour you want), but because the simple "get text from the
keyboard" function shouldn't conceal an eval of user text.

The solution, IMO, is to treat f-strings as expressions and NOT as
strings. They are no more strings than function calls are:

"###".join(os.system("sh"))

Everything that tokenizes Python code needs to be aware of the
different string prefixes already (eg a raw string literal doesn't end
at the same point as other string literals do), and this should be no
different. The stdlib ast.parse (really a wrapper around compile)
already gets this right:

>>> ast.dump(ast.parse('lambda x,y: f"{x} + {y} = {x+y}"'))
"Module(body=[Expr(value=Lambda(args=arguments(args=[arg(arg='x',
annotation=None), arg(arg='y', annotation=None)], vararg=None,
kwonlyargs=[], kw_defaults=[], kwarg=None, defaults=[]),
body=JoinedStr(values=[FormattedValue(value=Name(id='x', ctx=Load()),
conversion=-1, format_spec=None), Str(s=' + '),
FormattedValue(value=Name(id='y', ctx=Load()), conversion=-1,
format_spec=None), Str(s=' = '),
FormattedValue(value=BinOp(left=Name(id='x', ctx=Load()), op=Add(),
right=Name(id='y', ctx=Load())), conversion=-1,
format_spec=None)])))])"

So what is it that's trying to read something and is calling an
f-string a mere string?

ChrisA


From rosuav at gmail.com  Mon Nov 28 20:02:49 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 29 Nov 2016 12:02:49 +1100
Subject: Asyncio -- delayed calculation
In-Reply-To: <583cc9ea$0$22140$c3e8da3$5496439d@news.astraweb.com>
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CALwzidnTKni6SGB8Gpb0-zMdEkeqa1m6NN-ATfPkeY2YuNb1fg@mail.gmail.com>
 <mailman.38.1480348423.2073.python-list@python.org>
 <583cc9ea$0$22140$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPTjJmoMiZ4GciJgs7LO5vQHvKiRrSFDn5Cis-=4ro64-HP8GA@mail.gmail.com>

On Tue, Nov 29, 2016 at 11:20 AM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> On Tue, 29 Nov 2016 02:53 am, Ian Kelly wrote:
>
>> In order for the coroutines to actually do anything, you need to
>> schedule them in some way with the event loop. That could take the
>> form of awaiting them from some other coroutine, or passing them
>> directly to loop.run_until_complete or event_loop.create_task, or as
>> Chris suggested awaiting them as an aggregate.
>
> I thought that's what I had done, by calling
>
> loop.run_until_complete(main())

That's invoking a single task, main(). If you were to write this code
using threads, main would need to be thread-aware so that it can fork
appropriately - it needs to start new threads for the subtasks. Tasks
in asyncio are like cooperatively-switched threads or threadlets, and
if you want them to run in parallel, you have to instruct them to run
as separate tasks.

ChrisA


From steve+python at pearwood.info  Mon Nov 28 21:23:46 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Tue, 29 Nov 2016 13:23:46 +1100
Subject: Asyncio -- delayed calculation
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
Message-ID: <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>

On Tue, 29 Nov 2016 12:03 am, Chris Angelico wrote:

> On Mon, Nov 28, 2016 at 11:48 PM, Steve D'Aprano
> <steve+python at pearwood.info> wrote:
>> When I try running that, I get no output. No error, no exception, the
>> run_until_complete simply returns instantly.
> 
> When I do, I get this warning:
> 
> asynctest.py:17: RuntimeWarning: coroutine 'Counter.count_down' was
> never awaited
>   obj.count_down()

Ah yes, I saw that earlier, from an earlier version of the code that failed
with an exception. But it was enough to raise the warning, which then was
suppressed.


> Putting an 'await' in front of that call causes the tasks to be run
> consecutively, of course. The most similar code for running tasks
> concurrently seems to be this:
> 
> async def main():
>     pool = [Counter() for i in range(5)]
>     await asyncio.gather(*(obj.count_down() for obj in pool))
> 
> Taken from:
>
https://docs.python.org/3/library/asyncio-task.html#example-parallel-execution-of-tasks

This is confusing: why is this awaiting something inside an async function?
Doesn't that mean that the await asyncio.gather(...) call is turned
blocking?



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From zentraders at gmail.com  Mon Nov 28 21:27:12 2016
From: zentraders at gmail.com (Zentrader)
Date: Mon, 28 Nov 2016 18:27:12 -0800 (PST)
Subject: Asyncio -- delayed calculation
In-Reply-To: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <a3029f62-2295-4c7e-ba32-6d437ae5badf@googlegroups.com>

Take a look at Doug Hellmann's example using multiprocessing at https://pymotw.com/2/multiprocessing/basics.html  You should be able to substitute the count down example directly into the first example.


From rosuav at gmail.com  Mon Nov 28 22:21:24 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 29 Nov 2016 14:21:24 +1100
Subject: Asyncio -- delayed calculation
In-Reply-To: <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
 <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAPTjJmr_jjiYJn9Dsis+cjz9Wi==t+JS=jy5FYxwn4XcD2_uNw@mail.gmail.com>

On Tue, Nov 29, 2016 at 1:23 PM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> This is confusing: why is this awaiting something inside an async function?
> Doesn't that mean that the await asyncio.gather(...) call is turned
> blocking?

"await" means "don't continue this function until that's done". It
blocks the function until a non-blocking operation is done.

ChrisA


From zachary.ware+pylist at gmail.com  Mon Nov 28 22:32:05 2016
From: zachary.ware+pylist at gmail.com (Zachary Ware)
Date: Mon, 28 Nov 2016 21:32:05 -0600
Subject: Asyncio -- delayed calculation
In-Reply-To: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CAKJDb-Oraxt3UyR2NFY90kPTOqOPNR4qt88yEitsqKdekYzXVA@mail.gmail.com>

On Mon, Nov 28, 2016 at 6:48 AM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> What am I doing wrong?

Give yourself a bit more to debug with, since you're going to want to
do something with the result your expensive calculation anyway:

import asyncio

class Counter:
    def __init__(self, i):
        self.count = 10
        self.i = i

    async def count_down(self):
        print(self, self.i, "starting")
        while self.count > 0:
            # simulate a computation
            await asyncio.sleep(0.5)
            self.count -= 1
        print(self, self.i, "completed")
        return self.i + self.count

async def main():
    pool = [Counter(i) for i in range(5)]
    results = []
    for obj in pool:
        results.append(obj.count_down())
    return results

loop = asyncio.get_event_loop()
print(loop.run_until_complete(main()))


This gives you:

[<coroutine object Counter.count_down at 0x101631fc0>, <coroutine
object Counter.count_down at 0x101659048>, <coroutine object
Counter.count_down at 0x1016590f8>, <coroutine object
Counter.count_down at 0x101659150>, <coroutine object
Counter.count_down at 0x1016591a8>]
asynctest.py:25: RuntimeWarning: coroutine 'Counter.count_down' was
never awaited
  print(loop.run_until_complete(main()))

Ok, so let's fix that by adding an 'await' on line 21 (it's reported
at line 25 because that's when the unawaited coroutines are gc'd):

        results.append(await obj.count_down())

Running that gives:

<__main__.Counter object at 0x10203f978> 0 starting
<__main__.Counter object at 0x10203f978> 0 completed
<__main__.Counter object at 0x1025af710> 1 starting
<__main__.Counter object at 0x1025af710> 1 completed
<__main__.Counter object at 0x1025b60b8> 2 starting
<__main__.Counter object at 0x1025b60b8> 2 completed
<__main__.Counter object at 0x1025b60f0> 3 starting
<__main__.Counter object at 0x1025b60f0> 3 completed
<__main__.Counter object at 0x1025b6128> 4 starting
<__main__.Counter object at 0x1025b6128> 4 completed
[0, 1, 2, 3, 4]

Still not right, only one count_down is run at a time.  But that's
because we're using a synchronous for loop to await our results and
populate the results list.  Naively, I tried an 'async for', but
that's trying to be asynchronous in the wrong place:

Traceback (most recent call last):
  File "asynctest.py", line 25, in <module>
    print(loop.run_until_complete(main()))
  File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in
run_until_complete
    return future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "asynctest.py", line 20, in main
    async for obj in pool:
TypeError: 'async for' requires an object with __aiter__ method, got list

So instead, checking the docs suggests using asyncio.gather for
parallel execution of tasks, which takes a variable number of
'coros_or_futures'.  On the first attempt, we saw that we had
accidentally created a list of "coroutine objects", so lets go back
and use that:

--- asynctest.py.orig 2016-11-28 21:03:04.000000000 -0600
+++ asynctest.py 2016-11-28 21:03:35.000000000 -0600
@@ -16,9 +16,10 @@

 async def main():
     pool = [Counter(i) for i in range(5)]
-    results = []
+    coros = []
     for obj in pool:
-        results.append(await obj.count_down())
+        coros.append(obj.count_down())
+    results = asyncio.gather(*coros)
     return results

 loop = asyncio.get_event_loop()

Output:

<__main__.Counter object at 0x1026b6160> 4 starting
<__main__.Counter object at 0x10213f978> 0 starting
<__main__.Counter object at 0x1026b6128> 3 starting
<__main__.Counter object at 0x1026af748> 1 starting
<__main__.Counter object at 0x1026b60f0> 2 starting
<_GatheringFuture pending>

Now we've started everything asynchronously, but it exited way too
fast and never completed anything.  But instead of a list of results,
we got a _GatheringFuture at the end of everything.  So let's await
that:

    results = await asyncio.gather(*coros)

And now we get:

<__main__.Counter object at 0x101eb6160> 4 starting
<__main__.Counter object at 0x10063f978> 0 starting
<__main__.Counter object at 0x101eb6128> 3 starting
<__main__.Counter object at 0x101eaf748> 1 starting
<__main__.Counter object at 0x101eb60f0> 2 starting
<__main__.Counter object at 0x101eb6160> 4 completed
<__main__.Counter object at 0x10063f978> 0 completed
<__main__.Counter object at 0x101eb6128> 3 completed
<__main__.Counter object at 0x101eaf748> 1 completed
<__main__.Counter object at 0x101eb60f0> 2 completed
[0, 1, 2, 3, 4]


And there we have it.  Our final script is:

import asyncio

class Counter:
    def __init__(self, i):
        self.count = 10
        self.i = i

    async def count_down(self):
        print(self, self.i, "starting")
        while self.count > 0:
            # simulate a computation
            await asyncio.sleep(0.5)
            self.count -= 1
        print(self, self.i, "completed")
        return self.i + self.count

async def main():
    pool = [Counter(i) for i in range(5)]
    coros = []
    for obj in pool:
        coros.append(obj.count_down())
    results = await asyncio.gather(*coros)
    return results

loop = asyncio.get_event_loop()
print(loop.run_until_complete(main()))


I hope this is a nice prod to your understanding,
-- 
Zach


From greg.ewing at canterbury.ac.nz  Mon Nov 28 23:16:48 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Tue, 29 Nov 2016 17:16:48 +1300
Subject: Asyncio -- delayed calculation
In-Reply-To: <mailman.48.1480389686.2073.python-list@python.org>
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
 <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmr_jjiYJn9Dsis+cjz9Wi==t+JS=jy5FYxwn4XcD2_uNw@mail.gmail.com>
 <mailman.48.1480389686.2073.python-list@python.org>
Message-ID: <ea4dpiFlg8sU1@mid.individual.net>

Chris Angelico wrote:
> "await" means "don't continue this function until that's done". It
> blocks the function until a non-blocking operation is done.

However, *not* using 'await' doesn't mean the operation
will be done without blocking. Rather, it won't be done
at all (and is usually an error, but there's no way for
the implementation to detect it at the time).

I don't blame Steve for being confused. All the
terminology around async/await is inherently confusing
and counterintuitive, IMO. I'm disappointed that we've
ended up here.

-- 
Greg


From rosuav at gmail.com  Mon Nov 28 23:37:22 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 29 Nov 2016 15:37:22 +1100
Subject: Asyncio -- delayed calculation
In-Reply-To: <ea4dpiFlg8sU1@mid.individual.net>
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
 <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmr_jjiYJn9Dsis+cjz9Wi==t+JS=jy5FYxwn4XcD2_uNw@mail.gmail.com>
 <mailman.48.1480389686.2073.python-list@python.org>
 <ea4dpiFlg8sU1@mid.individual.net>
Message-ID: <CAPTjJmpdWpjR2FBSQYW89vjeyvC9ZH0MqO8L_gA4292O7LkTCA@mail.gmail.com>

On Tue, Nov 29, 2016 at 3:16 PM, Gregory Ewing
<greg.ewing at canterbury.ac.nz> wrote:
> Chris Angelico wrote:
>>
>> "await" means "don't continue this function until that's done". It
>> blocks the function until a non-blocking operation is done.
>
>
> However, *not* using 'await' doesn't mean the operation
> will be done without blocking. Rather, it won't be done
> at all (and is usually an error, but there's no way for
> the implementation to detect it at the time).
>
> I don't blame Steve for being confused. All the
> terminology around async/await is inherently confusing
> and counterintuitive, IMO. I'm disappointed that we've
> ended up here.

Asynchronous I/O is something to get your head around. As someone who
teaches both JavaScript and Python, I've run into this quite a bit,
and frankly, callbacks may be easy to explain at a concrete level, but
I'd much rather work with generator-based async functions (which JS is
getting soon too). The best way to think about it IMO is a "process"
that does blocking calls, and which the "system" can multitask away
from any time it's blocked. Your function awaits something, but Python
doesn't.

ChrisA


From no.email at nospam.invalid  Tue Nov 29 00:13:09 2016
From: no.email at nospam.invalid (Paul Rubin)
Date: Mon, 28 Nov 2016 21:13:09 -0800
Subject: Asyncio -- delayed calculation
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
 <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmr_jjiYJn9Dsis+cjz9Wi==t+JS=jy5FYxwn4XcD2_uNw@mail.gmail.com>
 <mailman.48.1480389686.2073.python-list@python.org>
 <ea4dpiFlg8sU1@mid.individual.net>
 <CAPTjJmpdWpjR2FBSQYW89vjeyvC9ZH0MqO8L_gA4292O7LkTCA@mail.gmail.com>
 <mailman.50.1480394245.2073.python-list@python.org>
Message-ID: <871sxu98d6.fsf@nightsong.com>

Chris Angelico <rosuav at gmail.com> writes:
> Asynchronous I/O is something to get your head around....  I'd much
> rather work with generator-based async functions...

I haven't gotten my head around Python asyncio and have been wanting
to read this:

   http://lucumr.pocoo.org/2016/10/30/i-dont-understand-asyncio/

Is that picture too bleak?  I've used traditional cooperative
multitasking systems in the past, and the main hazard with them is to be
careful to not run for too long without yielding.  Python's stuff seems
much more complicated, with weirder hazards, some stemming from the
hidden mutable state in every generator.  I wonder whether a
stackless-based green thread approach might have been simpler.


From nathan.ernst at gmail.com  Tue Nov 29 00:14:04 2016
From: nathan.ernst at gmail.com (Nathan Ernst)
Date: Mon, 28 Nov 2016 23:14:04 -0600
Subject: Asyncio -- delayed calculation
In-Reply-To: <CAPTjJmpdWpjR2FBSQYW89vjeyvC9ZH0MqO8L_gA4292O7LkTCA@mail.gmail.com>
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
 <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmr_jjiYJn9Dsis+cjz9Wi==t+JS=jy5FYxwn4XcD2_uNw@mail.gmail.com>
 <mailman.48.1480389686.2073.python-list@python.org>
 <ea4dpiFlg8sU1@mid.individual.net>
 <CAPTjJmpdWpjR2FBSQYW89vjeyvC9ZH0MqO8L_gA4292O7LkTCA@mail.gmail.com>
Message-ID: <CAPjBxg9R_g276oaSB_PhoY5NzUMyXiQjK83KL4AjmE5qYSWF9Q@mail.gmail.com>

To be fair, in other languages, such as C# or C++ with similar mechanisms,
if you don't ask for the result from an async or future task, there's no
guarantee the async task will be executed at all unless (or until) you ask
for the result. C++'s futures even give an explicit flag indicating you
want the task to be executed on the current thread, but deferred until the
result is asked for. I'm sure the people on the standards committee had a
rationale for this, but I see only marginal utility for that use case
(maybe such as a cancellable expensive calc?). Seems like "I might need the
result of this calc down the road, but I don't need it currently, so I'll
just kick the can down the road.". I'm not aware of a mechanism to make C#
behave that way, but I know it's possible that tasks will not execute until
they're asked for their result, but I don't know that it is 100%
deterministic.

To be honest, I think async/await stile programming is so new that there's
not a lot of good material about how to properly use it (as far as both
when and how). The limit of my experience has been in C# dealing with web
requests, where the usage is basically, I'm going to need this data, so
I'll fire off the async request for it, and I've got a lot of work I can do
before I need the result, so lets do that, until I need to wait (await) for
the result of that request before I can proceed any further. It can be
useful, but it is a new, completely different paradigm and methodology to
wrap ones head around. I've used it for about 2 years in C# land, and I
still don't know that I'm doing it right. I've not done any async/await in
Python space, but I'm eager to learn and try, when it may be appropriate,
But, most of my current use cases don't require it.

On Mon, Nov 28, 2016 at 10:37 PM, Chris Angelico <rosuav at gmail.com> wrote:

> On Tue, Nov 29, 2016 at 3:16 PM, Gregory Ewing
> <greg.ewing at canterbury.ac.nz> wrote:
> > Chris Angelico wrote:
> >>
> >> "await" means "don't continue this function until that's done". It
> >> blocks the function until a non-blocking operation is done.
> >
> >
> > However, *not* using 'await' doesn't mean the operation
> > will be done without blocking. Rather, it won't be done
> > at all (and is usually an error, but there's no way for
> > the implementation to detect it at the time).
> >
> > I don't blame Steve for being confused. All the
> > terminology around async/await is inherently confusing
> > and counterintuitive, IMO. I'm disappointed that we've
> > ended up here.
>
> Asynchronous I/O is something to get your head around. As someone who
> teaches both JavaScript and Python, I've run into this quite a bit,
> and frankly, callbacks may be easy to explain at a concrete level, but
> I'd much rather work with generator-based async functions (which JS is
> getting soon too). The best way to think about it IMO is a "process"
> that does blocking calls, and which the "system" can multitask away
> from any time it's blocked. Your function awaits something, but Python
> doesn't.
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From steve+comp.lang.python at pearwood.info  Tue Nov 29 00:31:31 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Tue, 29 Nov 2016 16:31:31 +1100
Subject: correct way to catch exception with Python 'with' statement
References: <CACT3xuX4J3maSiMgvzzSDG-xwJB2PKgyuUpMXbOO1-8mN+Bifg@mail.gmail.com>
 <mailman.33.1480313687.2073.python-list@python.org>
 <583be0f6$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <CACT3xuUKOEV44SEQ+8_VZKBNUG5d96SB0ztULKK=h1OdrGg5vw@mail.gmail.com>
 <mailman.36.1480346284.2073.python-list@python.org>
Message-ID: <583d12b4$0$2828$c3e8da3$76491128@news.astraweb.com>

On Tuesday 29 November 2016 02:18, Ganesh Pal wrote:

> On Mon, Nov 28, 2016 at 1:16 PM, Steven D'Aprano <
> steve+comp.lang.python at pearwood.info> wrote:
> 
>>
>>
>> There is no need to return True. The function either succeeds, or it
>> raises an
>> exception, so there is no need to return any value at all.
>>
>>
>  I returned True here ,because based on the result of this function , 

But the function *always* returns True, or it doesn't return at all: it raises.

Unless you have something like:

def func():
   do some work
   if condition:
      return False
   do more work
   return True

or similar, there's no point. When you write the documentation for the 
function, if it can only ever return True, then don't worry about returning 
True. Take the built-in methods as an example: dict.update either succeeds, or 
it raises an exception. It doesn't return True:

    # this is unnecessary
    flag = mydict.update(another_dict)
    if flag:
        print "update succeeded"
    else:
        print "update failed"


That cannot happen, because if the update fails, an exception is raised.

The bottom line is, since your function *only* has "return True" and doesn't 
have "return False" anywhere, there is no point to the "return True."


>  I would want to perform next steps
> 
>  Example
>                   if  create_files_append():
>                        do_somthing()
>                   else:
>                          do_next_thing()

That cannot happen. It either returns True, or it raises an exception, so the 
"else" clause will not be executed.




-- 
Steven
"Ever since I learned about confirmation bias, I've been seeing 
it everywhere." - Jon Ronson



From steve+comp.lang.python at pearwood.info  Tue Nov 29 00:32:44 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Tue, 29 Nov 2016 16:32:44 +1100
Subject: Asyncio -- delayed calculation
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
 <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmr_jjiYJn9Dsis+cjz9Wi==t+JS=jy5FYxwn4XcD2_uNw@mail.gmail.com>
 <mailman.48.1480389686.2073.python-list@python.org>
Message-ID: <583d12fc$0$2828$c3e8da3$76491128@news.astraweb.com>

On Tuesday 29 November 2016 14:21, Chris Angelico wrote:

> On Tue, Nov 29, 2016 at 1:23 PM, Steve D'Aprano
> <steve+python at pearwood.info> wrote:
>> This is confusing: why is this awaiting something inside an async function?
>> Doesn't that mean that the await asyncio.gather(...) call is turned
>> blocking?
> 
> "await" means "don't continue this function until that's done". It
> blocks the function until a non-blocking operation is done.

So that would be... "yes"?





-- 
Steven
"Ever since I learned about confirmation bias, I've been seeing 
it everywhere." - Jon Ronson



From rosuav at gmail.com  Tue Nov 29 00:42:44 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 29 Nov 2016 16:42:44 +1100
Subject: Asyncio -- delayed calculation
In-Reply-To: <871sxu98d6.fsf@nightsong.com>
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
 <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmr_jjiYJn9Dsis+cjz9Wi==t+JS=jy5FYxwn4XcD2_uNw@mail.gmail.com>
 <mailman.48.1480389686.2073.python-list@python.org>
 <ea4dpiFlg8sU1@mid.individual.net>
 <CAPTjJmpdWpjR2FBSQYW89vjeyvC9ZH0MqO8L_gA4292O7LkTCA@mail.gmail.com>
 <mailman.50.1480394245.2073.python-list@python.org>
 <871sxu98d6.fsf@nightsong.com>
Message-ID: <CAPTjJmpqrrTO-hkBfSgep-WYbE5Pi6F9utAtZb_CoMGCdyhU=w@mail.gmail.com>

On Tue, Nov 29, 2016 at 4:13 PM, Paul Rubin <no.email at nospam.invalid> wrote:
>
> I haven't gotten my head around Python asyncio and have been wanting
> to read this:
>
>    http://lucumr.pocoo.org/2016/10/30/i-dont-understand-asyncio/

It's talking a lot about how we got here, which isn't all necessary if
you just want to give asyncio a whirl. The conclusion at the end says
that you should just use 'async def' and not bother with all the older
forms, which I agree with (subject to the usual caveat that this
implies no support for older Pythons).

There's one thing that I really struggle with, though, and that's that
there's no easy and obvious way to demonstrate the lowest level of
operation. If "await x()" is like "yield from x()", how do you do the
innermost "yield" that actually does something? I have the same
confusion with Node.js, too. It's as if async primitives can't be
implemented in application code at all, they just have to be given to
you. Certainly it's not something made clear anywhere in the docs that
I've found.

ChrisA


From rosuav at gmail.com  Tue Nov 29 00:43:38 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 29 Nov 2016 16:43:38 +1100
Subject: Asyncio -- delayed calculation
In-Reply-To: <583d12fc$0$2828$c3e8da3$76491128@news.astraweb.com>
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
 <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmr_jjiYJn9Dsis+cjz9Wi==t+JS=jy5FYxwn4XcD2_uNw@mail.gmail.com>
 <mailman.48.1480389686.2073.python-list@python.org>
 <583d12fc$0$2828$c3e8da3$76491128@news.astraweb.com>
Message-ID: <CAPTjJmqGcKopT_F+QhYt_QX5oJwRc2qAY0VhPfBQqkv5XOuK-Q@mail.gmail.com>

On Tue, Nov 29, 2016 at 4:32 PM, Steven D'Aprano
<steve+comp.lang.python at pearwood.info> wrote:
> On Tuesday 29 November 2016 14:21, Chris Angelico wrote:
>
>> On Tue, Nov 29, 2016 at 1:23 PM, Steve D'Aprano
>> <steve+python at pearwood.info> wrote:
>>> This is confusing: why is this awaiting something inside an async function?
>>> Doesn't that mean that the await asyncio.gather(...) call is turned
>>> blocking?
>>
>> "await" means "don't continue this function until that's done". It
>> blocks the function until a non-blocking operation is done.
>
> So that would be... "yes"?

>From the point of view of the function, yes. From the point of view of
the rest of Python, no. It's a sign saying "Okay, Python, you can
alt-tab away from me now".

ChrisA


From ganesh1pal at gmail.com  Tue Nov 29 01:58:15 2016
From: ganesh1pal at gmail.com (Ganesh Pal)
Date: Tue, 29 Nov 2016 12:28:15 +0530
Subject: correct way to catch exception with Python 'with' statement
In-Reply-To: <583d12b4$0$2828$c3e8da3$76491128@news.astraweb.com>
References: <CACT3xuX4J3maSiMgvzzSDG-xwJB2PKgyuUpMXbOO1-8mN+Bifg@mail.gmail.com>
 <mailman.33.1480313687.2073.python-list@python.org>
 <583be0f6$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <CACT3xuUKOEV44SEQ+8_VZKBNUG5d96SB0ztULKK=h1OdrGg5vw@mail.gmail.com>
 <mailman.36.1480346284.2073.python-list@python.org>
 <583d12b4$0$2828$c3e8da3$76491128@news.astraweb.com>
Message-ID: <CACT3xuWavQYNZYPuKcyjotw+X5Uihk17ecQTT=VXq7tKGWdmgw@mail.gmail.com>

Thanks Steve I got what you were trying to explain  , nice learning  from
this conversation , what I was really doing wrong I had broken down my huge
 code into a simple program and had missed out returning False.

On Tue, Nov 29, 2016 at 11:01 AM, Steven D'Aprano <
steve+comp.lang.python at pearwood.info> wrote:

> On Tuesday 29 November 2016 02:18, Ganesh Pal wrote:
>
> > On Mon, Nov 28, 2016 at 1:16 PM, Steven D'Aprano <
> > steve+comp.lang.python at pearwood.info> wrote:
> >
> >>
> >>
> >> There is no need to return True. The function either succeeds, or it
> >> raises an
> >> exception, so there is no need to return any value at all.
> >>
> >>
> >  I returned True here ,because based on the result of this function ,
>
> But the function *always* returns True, or it doesn't return at all: it
> raises.
>
> Unless you have something like:
>
> def func():
>    do some work
>    if condition:
>       return False
>    do more work
>    return True
>
> or similar, there's no point. When you write the documentation for the
> function, if it can only ever return True, then don't worry about returning
> True. Take the built-in methods as an example: dict.update either
> succeeds, or
> it raises an exception. It doesn't return True:
>
>     # this is unnecessary
>     flag = mydict.update(another_dict)
>     if flag:
>         print "update succeeded"
>     else:
>         print "update failed"
>
>
> That cannot happen, because if the update fails, an exception is raised.
>
> The bottom line is, since your function *only* has "return True" and
> doesn't
> have "return False" anywhere, there is no point to the "return True."
>
>
> >  I would want to perform next steps
> >
> >  Example
> >                   if  create_files_append():
> >                        do_somthing()
> >                   else:
> >                          do_next_thing()
>
> That cannot happen. It either returns True, or it raises an exception, so
> the
> "else" clause will not be executed.
>
>
>
>
> --
> Steven
> "Ever since I learned about confirmation bias, I've been seeing
> it everywhere." - Jon Ronson
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From marko at pacujo.net  Tue Nov 29 02:20:01 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Tue, 29 Nov 2016 09:20:01 +0200
Subject: Asyncio -- delayed calculation
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
 <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmr_jjiYJn9Dsis+cjz9Wi==t+JS=jy5FYxwn4XcD2_uNw@mail.gmail.com>
 <mailman.48.1480389686.2073.python-list@python.org>
 <ea4dpiFlg8sU1@mid.individual.net>
Message-ID: <874m2qycpq.fsf@elektro.pacujo.net>

Gregory Ewing <greg.ewing at canterbury.ac.nz>:
> All the terminology around async/await is inherently confusing and
> counterintuitive, IMO. I'm disappointed that we've ended up here.

I think the conceptual mess can be clarified over time. Coroutines are
essentially threads. Why Python needs two threading implementations is
questionable. At least, with threads, you don't have to remember to tag
your function definitions with "async" and your functions calls with
"await".

Both threads and now coroutines were introduced as fig leafs to cover
the underlying finite state machines. Personally, I think it is a
mistake to hide the state machines. Instead, people should learn to
think in terms of state machines, which is the classic network protocol
model.


Marko


From marko at pacujo.net  Tue Nov 29 03:27:51 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Tue, 29 Nov 2016 10:27:51 +0200
Subject: correct way to catch exception with Python 'with' statement
References: <CACT3xuX4J3maSiMgvzzSDG-xwJB2PKgyuUpMXbOO1-8mN+Bifg@mail.gmail.com>
 <mailman.33.1480313687.2073.python-list@python.org>
 <583be0f6$0$22141$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <87zikiwv08.fsf@elektro.pacujo.net>

Steven D'Aprano <steve+comp.lang.python at pearwood.info>:
> There is no need to catch the exception if you're not going to do
> anything with it.

Correct. However, the question of the subject line is still a good one.
See:

    try:
        with open("xyz") as f:
            ...[A]...
    except FileNotFoundError:
        ...[B]...

The basic principle about exceptions is that if you are prepared to
catch one, catch it in the immediate vicinity of the operation that
might raise it. The code above might mistakenly catch an exception from
inside the "with" block.

You could then protect yourself like this:

   try:
       with open("xyz") as f:
           try:
               ...[A]...
           except FileNotFoundError as e:
               raise SmuggleItOutException(e)
   except FileNotFoundError:
       ...[B]...
   except SmuggleItOutException as e:
       raise e.exception_proper


However, I think the real answer is that you shouldn't mix the "with"
construct with exception handling. Instead you should write:

   try:
       f = open("xyz")
   except FileNotFoundError:
       ...[B]...
   try:
       ...[A]...
   finally:
       f.close()


Marko


From __peter__ at web.de  Tue Nov 29 04:06:33 2016
From: __peter__ at web.de (Peter Otten)
Date: Tue, 29 Nov 2016 10:06:33 +0100
Subject: correct way to catch exception with Python 'with' statement
References: <CACT3xuX4J3maSiMgvzzSDG-xwJB2PKgyuUpMXbOO1-8mN+Bifg@mail.gmail.com>
 <mailman.33.1480313687.2073.python-list@python.org>
 <583be0f6$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <87zikiwv08.fsf@elektro.pacujo.net>
Message-ID: <o1jgel$vh9$1@blaine.gmane.org>

Marko Rauhamaa wrote:

> However, I think the real answer is that you shouldn't mix the "with"
> construct with exception handling. Instead you should write:
> 
>    try:
>        f = open("xyz")
>    except FileNotFoundError:
>        ...[B]...
>    try:
>        ...[A]...
>    finally:
>        f.close()

What's the problem with spelling the above

try: 
    f = open(...)
except FileNotFoundError:
    ...
with f:
    ...

?



From marko at pacujo.net  Tue Nov 29 04:38:32 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Tue, 29 Nov 2016 11:38:32 +0200
Subject: correct way to catch exception with Python 'with' statement
References: <CACT3xuX4J3maSiMgvzzSDG-xwJB2PKgyuUpMXbOO1-8mN+Bifg@mail.gmail.com>
 <mailman.33.1480313687.2073.python-list@python.org>
 <583be0f6$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <87zikiwv08.fsf@elektro.pacujo.net> <o1jgel$vh9$1@blaine.gmane.org>
 <mailman.55.1480410402.2073.python-list@python.org>
Message-ID: <87h96q4odj.fsf@elektro.pacujo.net>

Peter Otten <__peter__ at web.de>:

> Marko Rauhamaa wrote:
>
>> However, I think the real answer is that you shouldn't mix the "with"
>> construct with exception handling. Instead you should write:
>> 
>>    try:
>>        f = open("xyz")
>>    except FileNotFoundError:
>>        ...[B]...
>>    try:
>>        ...[A]...
>>    finally:
>>        f.close()
>
> What's the problem with spelling the above
>
> try: 
>     f = open(...)
> except FileNotFoundError:
>     ...
> with f:
>     ...

Nothing.


Marko


From stephane at wirtel.be  Tue Nov 29 08:26:53 2016
From: stephane at wirtel.be (Stephane Wirtel)
Date: Tue, 29 Nov 2016 14:26:53 +0100
Subject: Last call for the Call For Proposals of PythonFOSDEM 2017
Message-ID: <20161129132653.e3vlya75shg2a7k2@sg1>

Because the deadline is imminent and because we have only received some
proposals, we will extend the current deadline. The new submission deadline is
2016-12-18.

Call For Proposals
==================

This is the official call for sessions for the Python devroom at FOSDEM 2017.

FOSDEM is the Free and Open source Software Developers' European Meeting, a free
and non-commercial two-day week-end that offers open source contributors a place
to meet, share ideas and collaborate.

It's the biggest event in Europe with +5000 hackers, +400 speakers.

For this edition, Python will be represented by its Community.
If you want to discuss with a lot of Python Users, it's the place to be!

Important dates
===============

* Submission deadlines: 2016-12-18
* Acceptance notifications: 2016-12-23

Practical
=========

* The duration for talks will be 30 minutes, including presentations and
questions and answers.
* Presentation can be recorded and streamed, sending your proposal implies
giving permission to be recorded.
* A mailing list for the Python devroom is available for discussions about
devroom organisation. You can register at this address:
https://lists.fosdem.org/listinfo/python-devroom

How to submit
=============

All submissions are made in the Pentabarf event planning tool at
https://penta.fosdem.org/submission/FOSDEM17

When submitting your talk in Pentabarf, make sure to select the Python devroom
as the Track.

Of course, if you already have a user account, please reuse it.

Questions
=========

Any questions, please send an email to info AT python-fosdem DOT org

Thank you for submitting your sessions and see you soon in Brussels to talk
about Python.

If you want to keep informed for this edition, you can follow our twitter
account @PythonFOSDEM.

* FOSDEM 2017: https://fosdem.org/2017
* Python Devroom: http://python-fosdem.org
* Twitter: https://twitter.com/PythonFOSDEM


Stephane

-- 
St?phane Wirtel - http://wirtel.be - @matrixise


From hemla21 at gmail.com  Tue Nov 29 09:17:57 2016
From: hemla21 at gmail.com (Heli)
Date: Tue, 29 Nov 2016 06:17:57 -0800 (PST)
Subject: best way to read a huge ascii file.
In-Reply-To: <5838bce2$0$22142$c3e8da3$5496439d@news.astraweb.com>
References: <eeed7603-4a21-4f7d-a484-5962bf44a30d@googlegroups.com>
 <5838bce2$0$22142$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <46dcc244-1058-4894-a1c4-2016ee010134@googlegroups.com>

Hi all, 

Let me update my question, I have an ascii file(7G) which has around 100M lines.  I read this file using : 

f=np.loadtxt(os.path.join(dir,myfile),delimiter=None,skiprows=0) 

x=f[:,1] 
y=f[:,2] 
z=f[:,3] 
id=f[:,0] 

I will need the x,y,z and id arrays later for interpolations. The problem is reading the file takes around 80 min while the interpolation only takes 15 mins.

I tried to get the memory increment used by each line of the script using python memory_profiler module.

The following line which reads the entire 7.4 GB file increments the memory usage by 3206.898 MiB (3.36 GB). First question is Why it does not increment the memory usage by 7.4 GB?

f=np.loadtxt(os.path.join(dir,myfile),delimiter=None,skiprows=0) 

The following 4 lines do not increment the memory at all. 
x=f[:,1] 
y=f[:,2] 
z=f[:,3] 
id=f[:,0] 

Finally I still would appreciate if you could recommend me what is the most optimized way to read/write to files in python? are numpy np.loadtxt and np.savetxt the best?

Thanks in Advance, 





From frank at chagford.com  Tue Nov 29 09:25:43 2016
From: frank at chagford.com (Frank Millman)
Date: Tue, 29 Nov 2016 16:25:43 +0200
Subject: async enumeration - possible?
Message-ID: <o1k355$da5$1@blaine.gmane.org>

Hi all

Python 3.6 has introduced Asynchronous Generators, which work very well.

Python 3.6.0b4 (default, Nov 22 2016, 05:30:12) [MSC v.1900 64 bit (AMD64)] 
on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> loop = asyncio.get_event_loop()
>>>
>>> async def counter(n):
...   for i in range(n):
...     yield i
...
>>> async def main():
...   c = counter(5)
...   async for j in c:
...     print(j)
...   print('done')
...
>>> loop.run_until_complete(main())
0
1
2
3
4
done

However, it does not allow you to enumerate over the generator output -

>>> async def main():
...   c = counter(5)
...   async for j, k in enumerate(c):
...     print(j, k)
...   print('done')
...
>>> loop.run_until_complete(main())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File 
"C:\Users\User\AppData\Local\Programs\Python\Python36\lib\asyncio\base_events.py", 
line 466, in run_until_complete
    return future.result()
TypeError: 'async_generator' object is not iterable
>>>

Is there any technical reason for this, or is it just that no-one has got 
around to writing an asynchronous version yet?

Frank Millman




From jussi.piitulainen at helsinki.fi  Tue Nov 29 09:45:23 2016
From: jussi.piitulainen at helsinki.fi (Jussi Piitulainen)
Date: Tue, 29 Nov 2016 16:45:23 +0200
Subject: best way to read a huge ascii file.
References: <eeed7603-4a21-4f7d-a484-5962bf44a30d@googlegroups.com>
 <5838bce2$0$22142$c3e8da3$5496439d@news.astraweb.com>
 <46dcc244-1058-4894-a1c4-2016ee010134@googlegroups.com>
Message-ID: <lf57f7me458.fsf@ling.helsinki.fi>

Heli writes:

> Hi all, 
>
> Let me update my question, I have an ascii file(7G) which has around
> 100M lines.  I read this file using :
>
> f=np.loadtxt(os.path.join(dir,myfile),delimiter=None,skiprows=0) 
>
> x=f[:,1] 
> y=f[:,2] 
> z=f[:,3] 
> id=f[:,0] 
>
> I will need the x,y,z and id arrays later for interpolations. The
> problem is reading the file takes around 80 min while the
> interpolation only takes 15 mins.

(Are there only those four columns in the file? I guess yes.)

> The following line which reads the entire 7.4 GB file increments the
> memory usage by 3206.898 MiB (3.36 GB). First question is Why it does
> not increment the memory usage by 7.4 GB?
>
> f=np.loadtxt(os.path.join(dir,myfile),delimiter=None,skiprows=0) 

In general, doubles take more space as text than as, well, doubles,
which (in those arrays) take eight bytes (64 bits) each:

>>> len("0.1411200080598672 -0.9899924966004454 -0.1425465430742778 20.085536923187668 ")
78
>>> 4*8
32

> Finally I still would appreciate if you could recommend me what is the
> most optimized way to read/write to files in python? are numpy
> np.loadtxt and np.savetxt the best?

A document I found says "This function aims to be a fast reader for
simply formatted files" so as long as you want to save the numbers as
text, this is probably meant to be the best way.

https://docs.scipy.org/doc/numpy/reference/generated/numpy.loadtxt.html

Perhaps there are binary load and save functions? They could be faster.
The binary data file would be opaque, but probably you are not editing
it by hand anyway.


From ian.g.kelly at gmail.com  Tue Nov 29 11:19:27 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Tue, 29 Nov 2016 09:19:27 -0700
Subject: async enumeration - possible?
In-Reply-To: <o1k355$da5$1@blaine.gmane.org>
References: <o1k355$da5$1@blaine.gmane.org>
Message-ID: <CALwzid=WVxc8xfQoU7Z=+h2At9R=rD48eoWuPw9NdL+o48nAzg@mail.gmail.com>

On Tue, Nov 29, 2016 at 7:25 AM, Frank Millman <frank at chagford.com> wrote:
> However, it does not allow you to enumerate over the generator output -
>
>>>> async def main():
>
> ...   c = counter(5)
> ...   async for j, k in enumerate(c):
> ...     print(j, k)
> ...   print('done')
> ...
>>>>
>>>> loop.run_until_complete(main())
>
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
>  File
> "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\asyncio\base_events.py",
> line 466, in run_until_complete
>    return future.result()
> TypeError: 'async_generator' object is not iterable
>>>>
>>>>
>
> Is there any technical reason for this, or is it just that no-one has got
> around to writing an asynchronous version yet?

No one has written an async enumerate afaik. You may be interested in
the aitertools package in PyPI:

https://pypi.python.org/pypi/aitertools/0.1.0

It also doesn't have an async enumerate. It does have an aiter
function that can wrap an ordinary iterable into an async iterable, so
one way to write that would be aiter(enumerate(c)), with the caveat
that the wrapped iterator is still running synchronously.

Otherwise, you can write one yourself. This doesn't support all the
features of enumerate, but it serves as demonstration:

    from itertools import count

    async def aenumerate(aiterable):
        counter = count()
        async for x in aiterable:
             yield next(counter), x


From marco.nawijn at colosso.nl  Tue Nov 29 11:43:38 2016
From: marco.nawijn at colosso.nl (marco.nawijn at colosso.nl)
Date: Tue, 29 Nov 2016 08:43:38 -0800 (PST)
Subject: best way to read a huge ascii file.
In-Reply-To: <46dcc244-1058-4894-a1c4-2016ee010134@googlegroups.com>
References: <eeed7603-4a21-4f7d-a484-5962bf44a30d@googlegroups.com>
 <5838bce2$0$22142$c3e8da3$5496439d@news.astraweb.com>
 <46dcc244-1058-4894-a1c4-2016ee010134@googlegroups.com>
Message-ID: <6fd22d04-d3e2-41b4-84a3-9b1cf7df177e@googlegroups.com>

On Tuesday, November 29, 2016 at 3:18:29 PM UTC+1, Heli wrote:
> Hi all, 
> 
> Let me update my question, I have an ascii file(7G) which has around 100M lines.  I read this file using : 
> 
> f=np.loadtxt(os.path.join(dir,myfile),delimiter=None,skiprows=0) 
> 
> x=f[:,1] 
> y=f[:,2] 
> z=f[:,3] 
> id=f[:,0] 
> 
> I will need the x,y,z and id arrays later for interpolations. The problem is reading the file takes around 80 min while the interpolation only takes 15 mins.
> 
> I tried to get the memory increment used by each line of the script using python memory_profiler module.
> 
> The following line which reads the entire 7.4 GB file increments the memory usage by 3206.898 MiB (3.36 GB). First question is Why it does not increment the memory usage by 7.4 GB?
> 
> f=np.loadtxt(os.path.join(dir,myfile),delimiter=None,skiprows=0) 
> 
> The following 4 lines do not increment the memory at all. 
> x=f[:,1] 
> y=f[:,2] 
> z=f[:,3] 
> id=f[:,0] 
> 
> Finally I still would appreciate if you could recommend me what is the most optimized way to read/write to files in python? are numpy np.loadtxt and np.savetxt the best?
> 
> Thanks in Advance,

Hi,

Have you considered storing the data in HDF5? There is an excellent Python
interface for this (see: http://www.h5py.org/). The advantage that you will
have is that no text to number conversion has to applied anymore. You can
directly operate on the datasets in the HDF5 database. 

If you would go this direction, the following would get you started:

>>> import h5py
>>> import numpy
>>> from numpy import uint32, float32, arange

>>> fd = h5py.File('demo.h5', mode='w') # Note that this will truncate
                                        # the file, use 'r' or 'a' if you
                                        # want to open an existing file
>>> observations = fd.create_group('/observations')
>>> N = 1000000 
>>> observations.create_dataset('id', data=arange(0, N, dtype=uint32))
>>> observations.create_dataset('x', data=numpy.random.random(N), dtype=float32)
>>> observations.create_dataset('y', data=numpy.random.random(N), dtype=float32)
>>> observations.create_dataset('z', data=numpy.random.random(N), dtype=float32)
>>> 
>>> fd.close()

Note that you can also combine x,y and z in a single dataset if you want to.
See the documentation for datasets for more information http://docs.h5py.org/en/latest/high/dataset.html

I would also advise you to carefully select the proper dtype for the arrays.
In particular if you know the value range for your datasets. This can save
you a lot of disk space and probably will increase the performance a little.

Marco


From bc at freeuk.com  Tue Nov 29 12:29:35 2016
From: bc at freeuk.com (BartC)
Date: Tue, 29 Nov 2016 17:29:35 +0000
Subject: best way to read a huge ascii file.
In-Reply-To: <46dcc244-1058-4894-a1c4-2016ee010134@googlegroups.com>
References: <eeed7603-4a21-4f7d-a484-5962bf44a30d@googlegroups.com>
 <5838bce2$0$22142$c3e8da3$5496439d@news.astraweb.com>
 <46dcc244-1058-4894-a1c4-2016ee010134@googlegroups.com>
Message-ID: <o1kdri$gnt$1@dont-email.me>

On 29/11/2016 14:17, Heli wrote:
> Hi all,
>
> Let me update my question, I have an ascii file(7G) which has around 100M lines.  I read this file using :
>
> f=np.loadtxt(os.path.join(dir,myfile),delimiter=None,skiprows=0)
>
> x=f[:,1]
> y=f[:,2]
> z=f[:,3]
> id=f[:,0]
>
> I will need the x,y,z and id arrays later for interpolations. The problem is reading the file takes around 80 min while the interpolation only takes 15 mins.
>
> I tried to get the memory increment used by each line of the script using python memory_profiler module.
>
> The following line which reads the entire 7.4 GB file increments the memory usage by 3206.898 MiB (3.36 GB). First question is Why it does not increment the memory usage by 7.4 GB?

Is there enough total RAM capacity for another 4.2GB?

But if the file is text, and being read into binary data in memory, it 
will be different. Usually binary data takes less space. I assume the 
loader doesn't load the entire text file first, do the conversions to 
binary, then unloads file, as that would then require 10.6GB during that 
process!

> f=np.loadtxt(os.path.join(dir,myfile),delimiter=None,skiprows=0)
>
> The following 4 lines do not increment the memory at all.
> x=f[:,1]
> y=f[:,2]
> z=f[:,3]
> id=f[:,0]

That's surprising because if those are slices, they would normally 
create a copy (I suppose you don't set f to 0 or something after those 
lines). But if numpy data is involved, I seem to remember that slices 
are actually views into the data.

> Finally I still would appreciate if you could recommend me what is the most optimized way to read/write to files in python? are numpy np.loadtxt and np.savetxt the best?

Why not post a sample couple of lines from the file? (We don't need the 
other 99,999,998 assuming they are all have the same format.) Then we 
can see if there's anything obviously inefficient about it.

-- 
Bartc


From ian.g.kelly at gmail.com  Tue Nov 29 13:41:45 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Tue, 29 Nov 2016 11:41:45 -0700
Subject: Asyncio -- delayed calculation
In-Reply-To: <CAPTjJmpqrrTO-hkBfSgep-WYbE5Pi6F9utAtZb_CoMGCdyhU=w@mail.gmail.com>
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
 <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmr_jjiYJn9Dsis+cjz9Wi==t+JS=jy5FYxwn4XcD2_uNw@mail.gmail.com>
 <mailman.48.1480389686.2073.python-list@python.org>
 <ea4dpiFlg8sU1@mid.individual.net>
 <CAPTjJmpdWpjR2FBSQYW89vjeyvC9ZH0MqO8L_gA4292O7LkTCA@mail.gmail.com>
 <mailman.50.1480394245.2073.python-list@python.org>
 <871sxu98d6.fsf@nightsong.com>
 <CAPTjJmpqrrTO-hkBfSgep-WYbE5Pi6F9utAtZb_CoMGCdyhU=w@mail.gmail.com>
Message-ID: <CALwzidkN+Xiuudr_pr+EUSM5EVeHz99YBedwdkKhF3+g9uzh8Q@mail.gmail.com>

On Mon, Nov 28, 2016 at 10:42 PM, Chris Angelico <rosuav at gmail.com> wrote:
> On Tue, Nov 29, 2016 at 4:13 PM, Paul Rubin <no.email at nospam.invalid> wrote:
>>
>> I haven't gotten my head around Python asyncio and have been wanting
>> to read this:
>>
>>    http://lucumr.pocoo.org/2016/10/30/i-dont-understand-asyncio/
>
> It's talking a lot about how we got here, which isn't all necessary if
> you just want to give asyncio a whirl. The conclusion at the end says
> that you should just use 'async def' and not bother with all the older
> forms, which I agree with (subject to the usual caveat that this
> implies no support for older Pythons).
>
> There's one thing that I really struggle with, though, and that's that
> there's no easy and obvious way to demonstrate the lowest level of
> operation. If "await x()" is like "yield from x()", how do you do the
> innermost "yield" that actually does something? I have the same
> confusion with Node.js, too. It's as if async primitives can't be
> implemented in application code at all, they just have to be given to
> you. Certainly it's not something made clear anywhere in the docs that
> I've found.

You mean how do you create something that can be awaited that doesn't
await something else in turn? With a Future.

import asyncio

class Awaitable(asyncio.Future):
  def wake_up_later(self):
    asyncio.get_event_loop().call_later(3, self.set_result, 42)

async def main():
  awaitable = Awaitable()
  awaitable.wake_up_later()
  print(await awaitable)

asyncio.get_event_loop().run_until_complete(main())

The trick is in arranging for the future's result to be set. For I/O
events you would typically do that by associating a callback with a
file descriptor on the event loop:

https://docs.python.org/3/library/asyncio-eventloop.html#watch-file-descriptors

If you need to do something particulary abstruse you can always write
a custom Selector or event loop:

https://docs.python.org/3/library/selectors.html#module-selectors
https://docs.python.org/3/library/asyncio-eventloop.html#base-event-loop


From handar94 at gmail.com  Tue Nov 29 14:35:26 2016
From: handar94 at gmail.com (handar94 at gmail.com)
Date: Tue, 29 Nov 2016 11:35:26 -0800 (PST)
Subject: csv into multiple columns using split function using python
Message-ID: <3a113795-fc6c-4952-821f-25d5afb168ee@googlegroups.com>

I am trying to split a specific column of csv into multiple column and then appending the split values at the end of each row.

`enter code here`
------------------------------------------------
import csv
fOpen1=open('Meta_D1.txt')

reader=csv.reader(fOpen1)
mylist=[elem[1].split(',') for elem in reader]
mylist1=[]

for elem in mylist1:
    mylist1.append(elem)


#writing to a csv file
with open('out1.csv', 'wb') as fp:
    myf = csv.writer(fp, delimiter=',')
    myf.writerows(mylist1)

---------------------------------------------------
Here is the link to file I am working on 2 column. https://spaces.hightail.com/space/4hFTj

Can someone guide me further?



From tjreedy at udel.edu  Tue Nov 29 14:49:06 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 29 Nov 2016 14:49:06 -0500
Subject: async enumeration - possible?
In-Reply-To: <o1k355$da5$1@blaine.gmane.org>
References: <o1k355$da5$1@blaine.gmane.org>
Message-ID: <o1km3g$ro8$1@blaine.gmane.org>

On 11/29/2016 9:25 AM, Frank Millman wrote:

> Is there any technical reason for this, or is it just that no-one has
> got around to writing an asynchronous version yet?

Google's first hit for 'aenumerate' is
https://pythonwise.blogspot.com/2015/11/aenumerate-enumerate-for-async-for.html

Note that updated 3.5.2+ code is in response to my comment.

-- 
Terry Jan Reedy



From marko at pacujo.net  Tue Nov 29 15:07:40 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Tue, 29 Nov 2016 22:07:40 +0200
Subject: async enumeration - possible?
References: <o1k355$da5$1@blaine.gmane.org> <o1km3g$ro8$1@blaine.gmane.org>
 <mailman.2.1480448964.17409.python-list@python.org>
Message-ID: <87mvgivylv.fsf@elektro.pacujo.net>

Terry Reedy <tjreedy at udel.edu>:

> On 11/29/2016 9:25 AM, Frank Millman wrote:
>
>> Is there any technical reason for this, or is it just that no-one has
>> got around to writing an asynchronous version yet?
>
> Google's first hit for 'aenumerate' is
> https://pythonwise.blogspot.com/2015/11/aenumerate-enumerate-for-async-for.html

Ok, so how about:

   aall(aiterable)
   aany(aiterable)
   class abytearray(aiterable[, encoding[, errors]])
   class adict(aiterable, **kwarg)
   class afilter(coro, aiterable)
   class afrozenset(aiterable)
   aiter(object[, sentinel])
   class alist(aiterable)
   amap(coro, aiterable, ...)
   amax(aiterable, *[, key, default])
   amin(aiterable, *[, key, default])
   anext(aiterator[, default])
   class aset(aiterable)
   asorted(aiterable[, key][, reverse])
   asum(aiterable[, start])
   atuple(aiterable)
   azip(*aiterables)

to name a few...

How about awaitable comprehensions?


Marko


From steve+python at pearwood.info  Tue Nov 29 18:20:01 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Wed, 30 Nov 2016 10:20:01 +1100
Subject: best way to read a huge ascii file.
References: <eeed7603-4a21-4f7d-a484-5962bf44a30d@googlegroups.com>
 <5838bce2$0$22142$c3e8da3$5496439d@news.astraweb.com>
 <46dcc244-1058-4894-a1c4-2016ee010134@googlegroups.com>
Message-ID: <583e0d23$0$1617$c3e8da3$5496439d@news.astraweb.com>

On Wed, 30 Nov 2016 01:17 am, Heli wrote:

> The following line which reads the entire 7.4 GB file increments the
> memory usage by 3206.898 MiB (3.36 GB). First question is Why it does not
> increment the memory usage by 7.4 GB?
>
> f=np.loadtxt(os.path.join(dir,myfile),delimiter=None,skiprows=0)

Floating point numbers as strings typically take up far more space than do
floats. On disk, a string like "3.141592653589793" requires 13 bytes. Plus
there are additional bytes used as separators between fields, and at least
one more byte (a newline) at the end of each record. Whereas, once
converted to a float (a C 64-bit double) it only requires 8 bytes. In a
numpy array, there's no separator needed and the values are tightly packed.

So its quite reasonable to expect a saving of around 50%.

> The following 4 lines do not increment the memory at all.
> x=f[:,1]
> y=f[:,2]
> z=f[:,3]
> id=f[:,0]

Numpy slices are views, not copies.


> Finally I still would appreciate if you could recommend me what is the
> most optimized way to read/write to files in python? are numpy np.loadtxt
> and np.savetxt the best?

You're not just reading a file. You're reading a file and converting
millions of strings to floats.

You are processing 7GB of data in 80 minutes, or around 1.5MB per second. Do
you have reason to think that's unreasonably slow? (Apart from wishing that
it were faster.) Where are you reading the file from? How much RAM do you
have?



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Tue Nov 29 18:21:57 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Wed, 30 Nov 2016 10:21:57 +1100
Subject: async enumeration - possible?
References: <o1k355$da5$1@blaine.gmane.org> <o1km3g$ro8$1@blaine.gmane.org>
 <mailman.2.1480448964.17409.python-list@python.org>
 <87mvgivylv.fsf@elektro.pacujo.net>
Message-ID: <583e0d96$0$1617$c3e8da3$5496439d@news.astraweb.com>

On Wed, 30 Nov 2016 07:07 am, Marko Rauhamaa wrote:

> Terry Reedy <tjreedy at udel.edu>:
> 
>> On 11/29/2016 9:25 AM, Frank Millman wrote:
>>
>>> Is there any technical reason for this, or is it just that no-one has
>>> got around to writing an asynchronous version yet?
>>
>> Google's first hit for 'aenumerate' is
>>
https://pythonwise.blogspot.com/2015/11/aenumerate-enumerate-for-async-for.html
> 
> Ok, so how about:
> 
>    aall(aiterable)
>    aany(aiterable)
>    class abytearray(aiterable[, encoding[, errors]])
[...]


What about them? What's your question?





-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Tue Nov 29 18:30:52 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Wed, 30 Nov 2016 10:30:52 +1100
Subject: Asyncio -- delayed calculation
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
 <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmr_jjiYJn9Dsis+cjz9Wi==t+JS=jy5FYxwn4XcD2_uNw@mail.gmail.com>
 <mailman.48.1480389686.2073.python-list@python.org>
 <ea4dpiFlg8sU1@mid.individual.net>
 <CAPTjJmpdWpjR2FBSQYW89vjeyvC9ZH0MqO8L_gA4292O7LkTCA@mail.gmail.com>
 <mailman.50.1480394245.2073.python-list@python.org>
 <871sxu98d6.fsf@nightsong.com>
 <CAPTjJmpqrrTO-hkBfSgep-WYbE5Pi6F9utAtZb_CoMGCdyhU=w@mail.gmail.com>
 <CALwzidkN+Xiuudr_pr+EUSM5EVeHz99YBedwdkKhF3+g9uzh8Q@mail.gmail.com>
 <mailman.1.1480444955.17409.python-list@python.org>
Message-ID: <583e0fae$0$1588$c3e8da3$5496439d@news.astraweb.com>

On Wed, 30 Nov 2016 05:41 am, Ian Kelly wrote:

> You mean how do you create something that can be awaited that doesn't
> await something else in turn? With a Future.
> 
> import asyncio
> 
> class Awaitable(asyncio.Future):
>   def wake_up_later(self):
>     asyncio.get_event_loop().call_later(3, self.set_result, 42)

Not to be confused with concurrent.Futures.

https://docs.python.org/3.5/library/concurrent.futures.html

https://docs.python.org/3.5/library/asyncio-task.html#asyncio.Future




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From woooee at gmail.com  Tue Nov 29 18:55:20 2016
From: woooee at gmail.com (woooee at gmail.com)
Date: Tue, 29 Nov 2016 15:55:20 -0800 (PST)
Subject: csv into multiple columns using split function using python
In-Reply-To: <3a113795-fc6c-4952-821f-25d5afb168ee@googlegroups.com>
References: <3a113795-fc6c-4952-821f-25d5afb168ee@googlegroups.com>
Message-ID: <b365904d-246d-4334-b0dd-569e8c127b5f@googlegroups.com>

Add some print statements to see what is happening, especially after the for elem in mylist1: statement


From paul.garcia2345 at gmail.com  Tue Nov 29 18:58:54 2016
From: paul.garcia2345 at gmail.com (paul.garcia2345 at gmail.com)
Date: Tue, 29 Nov 2016 15:58:54 -0800 (PST)
Subject: Python while loop
Message-ID: <0c642381-4dd2-48c5-bb22-b38f2d5b2b4a@googlegroups.com>

Write a program which prints the sum of numbers from 1 to 101 ( 1 and 101 are included) that are divisible by 5 (Use while loop)

This is the code: 

x=0
count=0
while x<=100:
    if x%5==0:
        count=count+x
    x=x+1
print(count)
        

Question: How does python know what count means ? I see that its declared. What i wanna know is how does it know the iteration or each number it finds divisible by 5 is the "Count" ??


From woooee at gmail.com  Tue Nov 29 18:59:59 2016
From: woooee at gmail.com (woooee at gmail.com)
Date: Tue, 29 Nov 2016 15:59:59 -0800 (PST)
Subject: correct way to catch exception with Python 'with' statement
In-Reply-To: <mailman.33.1480313687.2073.python-list@python.org>
References: <CACT3xuX4J3maSiMgvzzSDG-xwJB2PKgyuUpMXbOO1-8mN+Bifg@mail.gmail.com>
 <mailman.33.1480313687.2073.python-list@python.org>
Message-ID: <bea458ed-2e10-41f9-b223-2974f1e3633c@googlegroups.com>

If you want to do something only if the file exists (or does not), use os.path.isfile(filename)


From python at mrabarnett.plus.com  Tue Nov 29 19:46:47 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Wed, 30 Nov 2016 00:46:47 +0000
Subject: Python while loop
In-Reply-To: <0c642381-4dd2-48c5-bb22-b38f2d5b2b4a@googlegroups.com>
References: <0c642381-4dd2-48c5-bb22-b38f2d5b2b4a@googlegroups.com>
Message-ID: <bffa9f67-4d5b-ed60-c685-4bebc908fad5@mrabarnett.plus.com>

On 2016-11-29 23:58, paul.garcia2345 at gmail.com wrote:
> Write a program which prints the sum of numbers from 1 to 101 ( 1 and 101 are included) that are divisible by 5 (Use while loop)
>
> This is the code:
>
> x=0
> count=0
> while x<=100:
>     if x%5==0:
>         count=count+x
>     x=x+1
> print(count)
>
>
> Question: How does python know what count means ? I see that its declared. What i wanna know is how does it know the iteration or each number it finds divisible by 5 is the "Count" ??
>
It doesn't, it's just a name, and, anyway, in the code, 'x' is the count...

If it _did_ know (which is doesn't), wouldn't it be complaining that 
'count' isn't the count but the sum? :-)



From bc at freeuk.com  Tue Nov 29 20:03:05 2016
From: bc at freeuk.com (BartC)
Date: Wed, 30 Nov 2016 01:03:05 +0000
Subject: Python while loop
In-Reply-To: <0c642381-4dd2-48c5-bb22-b38f2d5b2b4a@googlegroups.com>
References: <0c642381-4dd2-48c5-bb22-b38f2d5b2b4a@googlegroups.com>
Message-ID: <o1l8dt$n7f$1@dont-email.me>

On 29/11/2016 23:58, paul.garcia2345 at gmail.com wrote:
> Write a program which prints the sum of numbers from 1 to 101 ( 1 and 101 are included) that are divisible by 5 (Use while loop)
>
> This is the code:
>
> x=0
> count=0
> while x<=100:
>     if x%5==0:
>         count=count+x
>     x=x+1
> print(count)

This looks at numbers from 0 to 100 inclusive, not 1 to 101. Although it 
doesn't affect the result. (It will add in 0 which is divisible by 5, 
but that doesn't change it either. If this is an exercise however, 
checking the correct range might be important!)

-- 
Bartc


From m at funkyhat.org  Tue Nov 29 20:17:38 2016
From: m at funkyhat.org (Matt Wheeler)
Date: Wed, 30 Nov 2016 01:17:38 +0000
Subject: correct way to catch exception with Python 'with' statement
In-Reply-To: <bea458ed-2e10-41f9-b223-2974f1e3633c@googlegroups.com>
References: <CACT3xuX4J3maSiMgvzzSDG-xwJB2PKgyuUpMXbOO1-8mN+Bifg@mail.gmail.com>
 <mailman.33.1480313687.2073.python-list@python.org>
 <bea458ed-2e10-41f9-b223-2974f1e3633c@googlegroups.com>
Message-ID: <CAG93HwFbHdi1D2SvrCoTRdoqEqZTFTSgwqtU6riVoq9EOCSRaQ@mail.gmail.com>

On Tue, 29 Nov 2016 at 23:59 <woooee at gmail.com> wrote:

> If you want to do something only if the file exists (or does not), use
> os.path.isfile(filename)
>

This opens you up to a potential race condition (and has potential security
implications, depending on the application), as you're using LBYL[0].
If you want to write robust code, you'll need to catch the
FileNotFoundError anyway, so unless you're doing many* file lookups and
expecting a large number of them* to not exist, you're probably better off
with EAFP[1] unless you're really sure (i.e. you've followed the 3 rules of
performance optimisation[2])


*I'm being quite vague here as you'd have to actually profile the
application to work out which way is quicker and if the difference is worth
worrying about.
Catching the exception in theory might be more expensive, but that cost
might be insignificant compared to the IO cost which you'll have either way.

[0] https://docs.python.org/2/glossary.html#term-lbyl
[1] https://docs.python.org/2/glossary.html#term-eafp
[2] http://wiki.c2.com/?RulesOfOptimization


From best_lay at yahoo.com  Tue Nov 29 21:20:42 2016
From: best_lay at yahoo.com (Wildman)
Date: Tue, 29 Nov 2016 20:20:42 -0600
Subject: Request Help With Byte/String Problem
Message-ID: <l4ydnbQLeNnnqqPFnZ2dnUU7-L_NnZ2d@giganews.com>

For the purpose of learning I am writing a script that will
return different information about the Linux machine where
it is running.  Sort of like the inxi utility.

Below is some code that I found that returns a list of the
network interface devices on the system.  It runs as is
perfectly on Python2 but I get the error pasted below the
code when run on Python3, the desired version.  I know it
has something to do with bytes vs. strings but I can' seem
to figure it out.  Any help appreciated.

import array
import socket
import struct

def all_interfaces():
    max_possible = 128  # arbitrary. raise if needed.
    bytes = max_possible * 32
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    names = array.array("B", '\0' * bytes)
    outbytes = struct.unpack('iL', fcntl.ioctl(
        s.fileno(),
        0x8912,  # SIOCGIFCONF
        struct.pack('iL', bytes, names.buffer_info()[0])
    ))[0]
    namestr = names.tostring()
    lst = []
    for i in range(0, outbytes, 40):
        name = namestr[i:i+16].split('\0', 1)[0]
        ip   = namestr[i+20:i+24]
        lst.append((name, ip))
    return lst

def format_ip(addr):
    return str(ord(addr[0])) + '.' + \
           str(ord(addr[1])) + '.' + \
           str(ord(addr[2])) + '.' + \
           str(ord(addr[3]))


ifs = all_interfaces()
for i in ifs:
    print("%12s   %s" % (i[0], format_ip(i[1])))


Traceback (most recent call last):
  File "./ifaces.py", line 32, in <module>
    ifs = all_interfaces()
  File "./ifaces.py", line 11, in all_interfaces
    names = array.array("B", '\0' * bytes)
TypeError: cannot use a str to initialize an array with typecode 'B'

-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From no.email at nospam.invalid  Tue Nov 29 21:29:51 2016
From: no.email at nospam.invalid (Paul Rubin)
Date: Tue, 29 Nov 2016 18:29:51 -0800
Subject: Request Help With Byte/String Problem
References: <l4ydnbQLeNnnqqPFnZ2dnUU7-L_NnZ2d@giganews.com>
Message-ID: <87inr57l9c.fsf@nightsong.com>

Wildman <best_lay at yahoo.com> writes:
>     names = array.array("B", '\0' * bytes)
> TypeError: cannot use a str to initialize an array with typecode 'B'

In Python 2, str is a byte string and you can do that.  In Python 3,
str is a unicode string, and if you want a byte string you have to
specify that explicitly, like b'foo' instead of 'foo'.  I.e.

     names = array.array("B", b'\0' * bytes)

should work.


From steve+comp.lang.python at pearwood.info  Tue Nov 29 22:20:31 2016
From: steve+comp.lang.python at pearwood.info (Steven D'Aprano)
Date: Wed, 30 Nov 2016 14:20:31 +1100
Subject: correct way to catch exception with Python 'with' statement
References: <CACT3xuX4J3maSiMgvzzSDG-xwJB2PKgyuUpMXbOO1-8mN+Bifg@mail.gmail.com>
 <mailman.33.1480313687.2073.python-list@python.org>
 <bea458ed-2e10-41f9-b223-2974f1e3633c@googlegroups.com>
Message-ID: <583e4580$0$1516$c3e8da3$5496439d@news.astraweb.com>

On Wednesday 30 November 2016 10:59, woooee at gmail.com wrote:

> If you want to do something only if the file exists (or does not), use
> os.path.isfile(filename)

No, don't do that. Just because the file exists, doesn't mean that you have 
permission to read or write to it.

Worse, the code is vulnerable to race conditions. Look at this:

if os.path.isfile(filename):
    with open(filename) as f:
        process(f)


Just because the file exists when you test it, doesn't mean it still exists a 
millisecond later when you go to open the file. On a modern multi-processing 
system, like Windows, OS X or Linux, a lot can happen in the microseconds 
between checking for the file's existence and actually accessing the file.

This is called a "Time Of Check To Time Of Use" bug, and it can be a security 
vulnerability.



-- 
Steven
"Ever since I learned about confirmation bias, I've been seeing 
it everywhere." - Jon Ronson



From rosuav at gmail.com  Tue Nov 29 22:22:07 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 30 Nov 2016 14:22:07 +1100
Subject: async enumeration - possible?
In-Reply-To: <87mvgivylv.fsf@elektro.pacujo.net>
References: <o1k355$da5$1@blaine.gmane.org> <o1km3g$ro8$1@blaine.gmane.org>
 <mailman.2.1480448964.17409.python-list@python.org>
 <87mvgivylv.fsf@elektro.pacujo.net>
Message-ID: <CAPTjJmq2kmCK9T__qgRdc6b3vQhc9EaEu6gaFuSqPzSOruCCYw@mail.gmail.com>

On Wed, Nov 30, 2016 at 7:07 AM, Marko Rauhamaa <marko at pacujo.net> wrote:
> Ok, so how about:
>
>    aall(aiterable)
>    aany(aiterable)
>    class abytearray(aiterable[, encoding[, errors]])
>    class adict(aiterable, **kwarg)
>    class afilter(coro, aiterable)
>    class afrozenset(aiterable)
>    aiter(object[, sentinel])
>    class alist(aiterable)
>    amap(coro, aiterable, ...)
>    amax(aiterable, *[, key, default])
>    amin(aiterable, *[, key, default])
>    anext(aiterator[, default])
>    class aset(aiterable)
>    asorted(aiterable[, key][, reverse])
>    asum(aiterable[, start])
>    atuple(aiterable)
>    azip(*aiterables)
>
> to name a few...
>
> How about awaitable comprehensions?

Any of these that depend on pumping the entire iterable can simply
synchronify [1] the iterable:

list(x async for x in aiterable)

Interestingly, I can't do that in a list comp:

>>> [x async for x in aiterable]
  File "<stdin>", line 1
    [x async for x in aiterable]
           ^
SyntaxError: invalid syntax

Not sure why. Anyhow. That removes the need for alist, atuple, amax,
etc. All you would need are the ones that process an iterable and
yield values individually, like azip and amap. Those can be provided
as recipes, third-party modules, or stdlib modules, until they prove
their worth as potential builtins.

ChrisA

[1] is that even a word?


From best_lay at yahoo.com  Tue Nov 29 23:01:51 2016
From: best_lay at yahoo.com (Wildman)
Date: Tue, 29 Nov 2016 22:01:51 -0600
Subject: Request Help With Byte/String Problem
References: <l4ydnbQLeNnnqqPFnZ2dnUU7-L_NnZ2d@giganews.com>
 <87inr57l9c.fsf@nightsong.com>
Message-ID: <uv2dnTnJH6Sy0qPFnZ2dnUU7-e2dnZ2d@giganews.com>

On Tue, 29 Nov 2016 18:29:51 -0800, Paul Rubin wrote:

> Wildman <best_lay at yahoo.com> writes:
>>     names = array.array("B", '\0' * bytes)
>> TypeError: cannot use a str to initialize an array with typecode 'B'
> 
> In Python 2, str is a byte string and you can do that.  In Python 3,
> str is a unicode string, and if you want a byte string you have to
> specify that explicitly, like b'foo' instead of 'foo'.  I.e.
> 
>      names = array.array("B", b'\0' * bytes)
> 
> should work.

I really appreciate your reply.  Your suggestion fixed that
problem, however, a new error appeared.  I am doing some
research to try to figure it out but no luck so far.

Traceback (most recent call last):
  File "./ifaces.py", line 33, in <module>
    ifs = all_interfaces()
  File "./ifaces.py", line 21, in all_interfaces
    name = namestr[i:i+16].split('\0', 1)[0]
TypeError: Type str doesn't support the buffer API


-- 
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!


From frank at chagford.com  Wed Nov 30 01:06:06 2016
From: frank at chagford.com (Frank Millman)
Date: Wed, 30 Nov 2016 08:06:06 +0200
Subject: async enumeration - possible?
In-Reply-To: <o1k355$da5$1@blaine.gmane.org>
References: <o1k355$da5$1@blaine.gmane.org>
Message-ID: <o1lq8d$so$1@blaine.gmane.org>

"Frank Millman"  wrote in message news:o1k355$da5$1 at blaine.gmane.org...
>
> Hi all
>
> Python 3.6 has introduced Asynchronous Generators, which work very well.
>
[...]
>
> However, it does not allow you to enumerate over the generator output -
>
[...]
>
> Is there any technical reason for this, or is it just that no-one has got 
> around to writing an asynchronous version yet?

Thanks for the replies.

@Ian
Thanks for your explanation and example.
The example is perfect for my simple requirement.

@Terry
I should have googled for aenumerate - didn't think of it.

However, this recipe is based on Python 3.5 and earlier.
Asynchronous Generators, introduced in 3.6, make it much easier.
See PEP 525 for the details -
    https://www.python.org/dev/peps/pep-0525/

@Chris
I agree, there is a big difference between functions which consume the 
entire iterable before returning the result, and those which behave 
asynchronously and return the value on-the-fly.

I happened upon enumerate. You mentioned zip and map, which are also likely 
to be useful in the right circumstances.

I did not quite follow your example, as I get the opposite result -

Python 3.6.0b4 (default, Nov 22 2016, 05:30:12) [MSC v.1900 64 bit (AMD64)] 
on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> loop = asyncio.get_event_loop()

>>> async def gen(n):
...   for i in range(n):
...     yield i
...
>>> async def main():
...   print([x async for x in gen(5)])
...
>>> loop.run_until_complete(main())
[0, 1, 2, 3, 4]

>>> async def main():
...   print(list(x async for x in gen(5)))
...
>>> loop.run_until_complete(main())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File 
"C:\Users\User\AppData\Local\Programs\Python\Python36\lib\asyncio\base_events.py", 
line 466, in run_until_complete
    return future.result()
TypeError: 'async_generator' object is not iterable
>>>

Frank




From marko at pacujo.net  Wed Nov 30 02:53:11 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Wed, 30 Nov 2016 09:53:11 +0200
Subject: async enumeration - possible?
References: <o1k355$da5$1@blaine.gmane.org> <o1km3g$ro8$1@blaine.gmane.org>
 <mailman.2.1480448964.17409.python-list@python.org>
 <87mvgivylv.fsf@elektro.pacujo.net>
 <CAPTjJmq2kmCK9T__qgRdc6b3vQhc9EaEu6gaFuSqPzSOruCCYw@mail.gmail.com>
 <mailman.3.1480476136.9583.python-list@python.org>
Message-ID: <87d1hd4d5k.fsf@elektro.pacujo.net>

Chris Angelico <rosuav at gmail.com>:

> On Wed, Nov 30, 2016 at 7:07 AM, Marko Rauhamaa <marko at pacujo.net> wrote:
> Any of these that depend on pumping the entire iterable can simply
> synchronify [1] the iterable:

One of the more useful ones might be:

    o = await anext(ait)

> list(x async for x in aiterable)
>
> Interestingly, I can't do that in a list comp:

I have a couple of points to make with my question:

 * We are seeing the reduplication of a large subset of Python's
   facilities. I really wonder if the coroutine fad is worth the price.

 * I don't think bulk iteration in asynchronous programming is ever that
   great of an idea. You want to be prepared for more than one possible
   stimulus in any given state. IOW, a state machine matrix might be
   sparse but it is never diagonal.


Marko


From ian.g.kelly at gmail.com  Wed Nov 30 03:10:37 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Wed, 30 Nov 2016 01:10:37 -0700
Subject: async enumeration - possible?
In-Reply-To: <CAPTjJmq2kmCK9T__qgRdc6b3vQhc9EaEu6gaFuSqPzSOruCCYw@mail.gmail.com>
References: <o1k355$da5$1@blaine.gmane.org> <o1km3g$ro8$1@blaine.gmane.org>
 <mailman.2.1480448964.17409.python-list@python.org>
 <87mvgivylv.fsf@elektro.pacujo.net>
 <CAPTjJmq2kmCK9T__qgRdc6b3vQhc9EaEu6gaFuSqPzSOruCCYw@mail.gmail.com>
Message-ID: <CALwzidnv7P-nO8vDQyKs2jfS_yCo7HZM+hdrV2smLcacnX9iiA@mail.gmail.com>

On Tue, Nov 29, 2016 at 8:22 PM, Chris Angelico <rosuav at gmail.com> wrote:
> Interestingly, I can't do that in a list comp:
>
>>>> [x async for x in aiterable]
>   File "<stdin>", line 1
>     [x async for x in aiterable]
>            ^
> SyntaxError: invalid syntax
>
> Not sure why.

Because you tried to use an async comprehension outside of a coroutine.

py> [x async for x in y]
  File "<stdin>", line 1
    [x async for x in y]
           ^
SyntaxError: invalid syntax
py> async def foo():
...     [x async for x in y]
...

The same is true for async generator expressions. The documentation is
clear that this is illegal for the async for statement:

https://docs.python.org/3.6/reference/compound_stmts.html#the-async-for-statement

I don't see anything about async comprehensions or async generators
outside of the "what's new" section, but it stands to reason that the
same would apply.

On Tue, Nov 29, 2016 at 11:06 PM, Frank Millman <frank at chagford.com> wrote:
>>>> async def main():
>
> ...   print(list(x async for x in gen(5)))
>
> ...
>>>>
>>>> loop.run_until_complete(main())
>
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
>  File
> "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\asyncio\base_events.py",
> line 466, in run_until_complete
>    return future.result()
> TypeError: 'async_generator' object is not iterable

Yeah, that's what I would expect. (x async for x in foo) is
essentially a no-op, just like its synchronous equivalent; it takes an
asynchronous iterator and produces an equivalent asynchronous
iterator. Meanwhile, list() can't consume an async iterator because
the list constructor isn't a coroutine. I don't think it's generally
possible to "synchronify" an async iterator other than to materialize
it. E.g.:

def alist(aiterable):
    result = []
    async for value in aiterable:
        result.append(value)
    return result

And I find it a little disturbing that I actually can't see a better
way to build a list from an async iterator than that.


From rosuav at gmail.com  Wed Nov 30 03:20:15 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 30 Nov 2016 19:20:15 +1100
Subject: async enumeration - possible?
In-Reply-To: <CALwzidnv7P-nO8vDQyKs2jfS_yCo7HZM+hdrV2smLcacnX9iiA@mail.gmail.com>
References: <o1k355$da5$1@blaine.gmane.org> <o1km3g$ro8$1@blaine.gmane.org>
 <mailman.2.1480448964.17409.python-list@python.org>
 <87mvgivylv.fsf@elektro.pacujo.net>
 <CAPTjJmq2kmCK9T__qgRdc6b3vQhc9EaEu6gaFuSqPzSOruCCYw@mail.gmail.com>
 <CALwzidnv7P-nO8vDQyKs2jfS_yCo7HZM+hdrV2smLcacnX9iiA@mail.gmail.com>
Message-ID: <CAPTjJmqHemdhxt_R3CkLNwth=9QArHnSwYpxxodNNH6ZZ4MNVg@mail.gmail.com>

On Wed, Nov 30, 2016 at 7:10 PM, Ian Kelly <ian.g.kelly at gmail.com> wrote:
> On Tue, Nov 29, 2016 at 8:22 PM, Chris Angelico <rosuav at gmail.com> wrote:
>> Interestingly, I can't do that in a list comp:
>>
>>>>> [x async for x in aiterable]
>>   File "<stdin>", line 1
>>     [x async for x in aiterable]
>>            ^
>> SyntaxError: invalid syntax
>>
>> Not sure why.
>
> Because you tried to use an async comprehension outside of a coroutine.
>
> py> [x async for x in y]
>   File "<stdin>", line 1
>     [x async for x in y]
>            ^
> SyntaxError: invalid syntax
> py> async def foo():
> ...     [x async for x in y]
> ...
>
> The same is true for async generator expressions. The documentation is
> clear that this is illegal for the async for statement:
>
> https://docs.python.org/3.6/reference/compound_stmts.html#the-async-for-statement

Hmm. The thing is, comprehensions and generators are implemented with
their own nested functions. So I would expect that their use of async
is independent of the function they're in. But maybe we have a bug
here?

>>> async def spam():
...     def ham():
...         async for i in x:
...             pass
...
>>> def ham():
...     async for i in x:
  File "<stdin>", line 2
    async for i in x:
            ^
SyntaxError: invalid syntax
>>> def ham():
...     async def spam():
...         async for i in x:
...             pass
...
>>>

Clearly the second one is correct to throw SyntaxError, and the third
is correctly acceptable. But the first one, ISTM, should be an error
too.

> Yeah, that's what I would expect. (x async for x in foo) is
> essentially a no-op, just like its synchronous equivalent; it takes an
> asynchronous iterator and produces an equivalent asynchronous
> iterator. Meanwhile, list() can't consume an async iterator because
> the list constructor isn't a coroutine. I don't think it's generally
> possible to "synchronify" an async iterator other than to materialize
> it. E.g.:
>
> def alist(aiterable):
>     result = []
>     async for value in aiterable:
>         result.append(value)
>     return result
>
> And I find it a little disturbing that I actually can't see a better
> way to build a list from an async iterator than that.

Oh. Oops. That materialization was exactly what I intended to happen
with the comprehension. Problem: Your version doesn't work either,
although I think it probably _does_ work if you declare that as "async
def alist".

Shows you just how well I understand Python's asyncness, doesn't it?

ChrisA


From ian.g.kelly at gmail.com  Wed Nov 30 03:25:19 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Wed, 30 Nov 2016 01:25:19 -0700
Subject: async enumeration - possible?
In-Reply-To: <87d1hd4d5k.fsf@elektro.pacujo.net>
References: <o1k355$da5$1@blaine.gmane.org> <o1km3g$ro8$1@blaine.gmane.org>
 <mailman.2.1480448964.17409.python-list@python.org>
 <87mvgivylv.fsf@elektro.pacujo.net>
 <CAPTjJmq2kmCK9T__qgRdc6b3vQhc9EaEu6gaFuSqPzSOruCCYw@mail.gmail.com>
 <mailman.3.1480476136.9583.python-list@python.org>
 <87d1hd4d5k.fsf@elektro.pacujo.net>
Message-ID: <CALwzid=tQCZ4q7bxAgvK9SOrN2vbGR5_5eVvGQySJgHEeFWPUw@mail.gmail.com>

On Wed, Nov 30, 2016 at 12:53 AM, Marko Rauhamaa <marko at pacujo.net> wrote:
> I have a couple of points to make with my question:
>
>  * We are seeing the reduplication of a large subset of Python's
>    facilities. I really wonder if the coroutine fad is worth the price.

I don't think there's any technical reason why functions like zip
can't support both synchronous and asynchronous iterators. For
example:

_zip = __builtins__.zip
_azip = (some asynchronous zip implementation)

class zip:
    def __init__(self, *args):
        self._args = args
    def __iter__(self):
        return _zip(*self._args)
    def __aiter__(self):
        return _azip(*self._args)

Now I can do "for x, y in zip(a, b)" or "async for x, y in
zip(async_a, async_b)" and either will work as expected. Of course, if
you use the wrong construct then you'll probably get an error since
the underlying iterators don't support the protocol. But it keeps the
builtin namespace simple and clean.


From frank at chagford.com  Wed Nov 30 03:29:14 2016
From: frank at chagford.com (Frank Millman)
Date: Wed, 30 Nov 2016 10:29:14 +0200
Subject: async enumeration - possible?
In-Reply-To: <87d1hd4d5k.fsf@elektro.pacujo.net>
References: <o1k355$da5$1@blaine.gmane.org> <o1km3g$ro8$1@blaine.gmane.org>
 <mailman.2.1480448964.17409.python-list@python.org>
 <87mvgivylv.fsf@elektro.pacujo.net>
 <CAPTjJmq2kmCK9T__qgRdc6b3vQhc9EaEu6gaFuSqPzSOruCCYw@mail.gmail.com>
 <mailman.3.1480476136.9583.python-list@python.org>
 <87d1hd4d5k.fsf@elektro.pacujo.net>
Message-ID: <o1m2kp$7jp$1@blaine.gmane.org>

"Marko Rauhamaa"  wrote in message news:87d1hd4d5k.fsf at elektro.pacujo.net...
>
> One of the more useful ones might be:
>
>     o = await anext(ait)
>

Definitely!

But I found it easy to write my own -

async def anext(aiter):
    return await aiter.__anext__()

[...]

>   I don't think bulk iteration in asynchronous programming is ever that
>   great of an idea. You want to be prepared for more than one possible
>   stimulus in any given state. IOW, a state machine matrix might be
>   sparse but it is never diagonal.

I am not familiar with your terminology here, so my comment may be way 
off-track.

I use 'bulk iteration' a lot in my app. It is a client/server multi-user 
business/accounting app.

If a user wants to view the contents of a large table, or I want to print 
statements for a lot of customers,  I can request the data and process it as 
it arrives, without blocking the other users.

I find that very powerful.

Frank




From ian.g.kelly at gmail.com  Wed Nov 30 03:39:42 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Wed, 30 Nov 2016 01:39:42 -0700
Subject: async enumeration - possible?
In-Reply-To: <CAPTjJmqHemdhxt_R3CkLNwth=9QArHnSwYpxxodNNH6ZZ4MNVg@mail.gmail.com>
References: <o1k355$da5$1@blaine.gmane.org> <o1km3g$ro8$1@blaine.gmane.org>
 <mailman.2.1480448964.17409.python-list@python.org>
 <87mvgivylv.fsf@elektro.pacujo.net>
 <CAPTjJmq2kmCK9T__qgRdc6b3vQhc9EaEu6gaFuSqPzSOruCCYw@mail.gmail.com>
 <CALwzidnv7P-nO8vDQyKs2jfS_yCo7HZM+hdrV2smLcacnX9iiA@mail.gmail.com>
 <CAPTjJmqHemdhxt_R3CkLNwth=9QArHnSwYpxxodNNH6ZZ4MNVg@mail.gmail.com>
Message-ID: <CALwzidnje1KBmhAqvGKE7FY4W_K=XFGhc9BohbWqHGSGEULsqA@mail.gmail.com>

On Wed, Nov 30, 2016 at 1:20 AM, Chris Angelico <rosuav at gmail.com> wrote:
> Hmm. The thing is, comprehensions and generators are implemented with
> their own nested functions. So I would expect that their use of async
> is independent of the function they're in. But maybe we have a bug
> here?
>
>>>> async def spam():
> ...     def ham():
> ...         async for i in x:
> ...             pass
> ...
>>>> def ham():
> ...     async for i in x:
>   File "<stdin>", line 2
>     async for i in x:
>             ^
> SyntaxError: invalid syntax
>>>> def ham():
> ...     async def spam():
> ...         async for i in x:
> ...             pass
> ...
>>>>
>
> Clearly the second one is correct to throw SyntaxError, and the third
> is correctly acceptable. But the first one, ISTM, should be an error
> too.

Yeah, that looks like a bug to me. Note that 'await' results in a
clear error in the first case:

>>> async def ham():
...     def spam():
...         await foo
...
  File "<stdin>", line 3
SyntaxError: 'await' outside async function


>> Yeah, that's what I would expect. (x async for x in foo) is
>> essentially a no-op, just like its synchronous equivalent; it takes an
>> asynchronous iterator and produces an equivalent asynchronous
>> iterator. Meanwhile, list() can't consume an async iterator because
>> the list constructor isn't a coroutine. I don't think it's generally
>> possible to "synchronify" an async iterator other than to materialize
>> it. E.g.:
>>
>> def alist(aiterable):
>>     result = []
>>     async for value in aiterable:
>>         result.append(value)
>>     return result
>>
>> And I find it a little disturbing that I actually can't see a better
>> way to build a list from an async iterator than that.
>
> Oh. Oops. That materialization was exactly what I intended to happen
> with the comprehension. Problem: Your version doesn't work either,
> although I think it probably _does_ work if you declare that as "async
> def alist".

Yes, that's what I meant.


From ian.g.kelly at gmail.com  Wed Nov 30 03:51:29 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Wed, 30 Nov 2016 01:51:29 -0700
Subject: async enumeration - possible?
In-Reply-To: <o1m2kp$7jp$1@blaine.gmane.org>
References: <o1k355$da5$1@blaine.gmane.org> <o1km3g$ro8$1@blaine.gmane.org>
 <mailman.2.1480448964.17409.python-list@python.org>
 <87mvgivylv.fsf@elektro.pacujo.net>
 <CAPTjJmq2kmCK9T__qgRdc6b3vQhc9EaEu6gaFuSqPzSOruCCYw@mail.gmail.com>
 <mailman.3.1480476136.9583.python-list@python.org>
 <87d1hd4d5k.fsf@elektro.pacujo.net> <o1m2kp$7jp$1@blaine.gmane.org>
Message-ID: <CALwzid=HriJTV4P1_6FRKQUb25-O1i8oUQuxOzd+auJGL7+LRw@mail.gmail.com>

On Wed, Nov 30, 2016 at 1:29 AM, Frank Millman <frank at chagford.com> wrote:
> "Marko Rauhamaa"  wrote in message news:87d1hd4d5k.fsf at elektro.pacujo.net...
>>
>>
>> One of the more useful ones might be:
>>
>>     o = await anext(ait)
>>
>
> Definitely!
>
> But I found it easy to write my own -
>
> async def anext(aiter):
>    return await aiter.__anext__()

Even simpler:

def anext(aiter):
    return aiter.__anext__()

As a general rule, if the only await in a coroutine is immediately
prior to the return, then it doesn't need to be a coroutine. Just
return the thing it's awaiting so that the caller can be rid of the
middle man and await it directly.


From __peter__ at web.de  Wed Nov 30 04:14:18 2016
From: __peter__ at web.de (Peter Otten)
Date: Wed, 30 Nov 2016 10:14:18 +0100
Subject: csv into multiple columns using split function using python
References: <3a113795-fc6c-4952-821f-25d5afb168ee@googlegroups.com>
Message-ID: <o1m598$m30$1@blaine.gmane.org>

handar94 at gmail.com wrote:

> I am trying to split a specific column of csv into multiple column and
> then appending the split values at the end of each row.
> 
> `enter code here`
> ------------------------------------------------
> import csv
> fOpen1=open('Meta_D1.txt')
> 
> reader=csv.reader(fOpen1)
> mylist=[elem[1].split(',') for elem in reader]
> mylist1=[]
> 
> for elem in mylist1:
>     mylist1.append(elem)
> 
> 
> #writing to a csv file
> with open('out1.csv', 'wb') as fp:
>     myf = csv.writer(fp, delimiter=',')
>     myf.writerows(mylist1)
> 
> ---------------------------------------------------
> Here is the link to file I am working on 2 column.
> https://spaces.hightail.com/space/4hFTj
> 
> Can someone guide me further?

Use helper functions to process one row and the column you want to split:

import csv

def split_column(column):
    """
    >>> split_column("foo,bar,baz")
    ['foo', 'bar', 'baz']
    """
    return column.split(",")

def process_row(row):
    """
    >>> process_row(["foo", "one,two,three", "bar"])
    ['foo', 'one,two,three', 'bar', 'one', 'two', 'three']
    """
    new_row = row + split_column(row[1])
    return new_row

def convert_csv(infile, outfile):
    with open(infile) as instream:
        rows = csv.reader(instream)
        with open(outfile, "w") as outstream:
            writer = csv.writer(outstream, delimiter=",")
            writer.writerows(process_row(row) for row in rows)

if __name__ == "__main__":
    convert_csv(infile="infile.csv", outfile="outfile.csv")

That makes it easy to identify (and fix) the parts that do not work to your 
satisfaction. Let's say you want to remove the original unsplit second 
column. You know you only have to modify process_row(). You change the 
doctest first

def process_row(row):
    """
    >>> process_row(["foo", "one,two,three", "bar"])
    ['foo', 'bar', 'one', 'two', 'three']
    """
    new_row = row + split_column(row[1])
    return new_row

and verify that it fails:

$ python3 -m doctest split_column2.py
**********************************************************************
File "/somewhere/split_column2.py", line 12, in split_column2.process_row
Failed example:
    process_row(["foo", "one,two,three", "bar"])
Expected:
    ['foo', 'bar', 'one', 'two', 'three']
Got:
    ['foo', 'one,two,three', 'bar', 'one', 'two', 'three']
**********************************************************************
1 items had failures:
   1 of   1 in split_column2.process_row
***Test Failed*** 1 failures.

Then fix the function until you get

$ python3 -m doctest split_column2.py

If you don't trust the "no output means everything is OK" philosophy use the 
--verbose flag:

$ python3 -m doctest --verbose split_column2.py
Trying:
    process_row(["foo", "one,two,three", "bar"])
Expecting:
    ['foo', 'bar', 'one', 'two', 'three']
ok
Trying:
    split_column("foo,bar,baz")
Expecting:
    ['foo', 'bar', 'baz']
ok
2 items had no tests:
    split_column2
    split_column2.convert_csv
2 items passed all tests:
   1 tests in split_column2.process_row
   1 tests in split_column2.split_column
2 tests in 4 items.
2 passed and 0 failed.
Test passed.




From frank at chagford.com  Wed Nov 30 04:14:19 2016
From: frank at chagford.com (Frank Millman)
Date: Wed, 30 Nov 2016 11:14:19 +0200
Subject: async enumeration - possible?
In-Reply-To: <CALwzid=HriJTV4P1_6FRKQUb25-O1i8oUQuxOzd+auJGL7+LRw@mail.gmail.com>
References: <o1k355$da5$1@blaine.gmane.org> <o1km3g$ro8$1@blaine.gmane.org>
 <mailman.2.1480448964.17409.python-list@python.org>
 <87mvgivylv.fsf@elektro.pacujo.net>
 <CAPTjJmq2kmCK9T__qgRdc6b3vQhc9EaEu6gaFuSqPzSOruCCYw@mail.gmail.com>
 <mailman.3.1480476136.9583.python-list@python.org>
 <87d1hd4d5k.fsf@elektro.pacujo.net> <o1m2kp$7jp$1@blaine.gmane.org>
 <CALwzid=HriJTV4P1_6FRKQUb25-O1i8oUQuxOzd+auJGL7+LRw@mail.gmail.com>
Message-ID: <o1m599$min$1@blaine.gmane.org>

"Ian Kelly"  wrote in message 
news:CALwzid=HriJTV4P1_6FRKQUb25-O1i8oUQuxOzd+auJGL7+LRw at mail.gmail.com...
>
> On Wed, Nov 30, 2016 at 1:29 AM, Frank Millman <frank at chagford.com> wrote:
> >
> > async def anext(aiter):
> >    return await aiter.__anext__()
>
> Even simpler:
>
> def anext(aiter):
>     return aiter.__anext__()
>
> As a general rule, if the only await in a coroutine is immediately
> prior to the return, then it doesn't need to be a coroutine. Just
> return the thing it's awaiting so that the caller can be rid of the
> middle man and await it directly.

Fascinating!

Now I will have to go through all my code looking for similar occurrences. I 
am sure I will find some!

Frank




From __peter__ at web.de  Wed Nov 30 04:27:53 2016
From: __peter__ at web.de (Peter Otten)
Date: Wed, 30 Nov 2016 10:27:53 +0100
Subject: async enumeration - possible?
References: <o1k355$da5$1@blaine.gmane.org> <o1km3g$ro8$1@blaine.gmane.org>
 <mailman.2.1480448964.17409.python-list@python.org>
 <87mvgivylv.fsf@elektro.pacujo.net>
 <583e0d96$0$1617$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <o1m62l$caa$1@blaine.gmane.org>

Steve D'Aprano wrote:

> On Wed, 30 Nov 2016 07:07 am, Marko Rauhamaa wrote:
> 
>> Terry Reedy <tjreedy at udel.edu>:
>> 
>>> On 11/29/2016 9:25 AM, Frank Millman wrote:
>>>
>>>> Is there any technical reason for this, or is it just that no-one has
>>>> got around to writing an asynchronous version yet?
>>>
>>> Google's first hit for 'aenumerate' is
>>>
> https://pythonwise.blogspot.com/2015/11/aenumerate-enumerate-for-async-for.html
>> 
>> Ok, so how about:
>> 
>>    aall(aiterable)
>>    aany(aiterable)
>>    class abytearray(aiterable[, encoding[, errors]])
> [...]
> 
> 
> What about them? What's your question?

Well, my questions as someone who hasn't touched the async stuff so far 
would be:

Is there a viable approach to provide

(a) support for async without duplicating the stdlib
(b) a reasonably elegant way to access the async versions

I hope we can agree that prepending an "a" to the name should be the last 
resort.



From marko at pacujo.net  Wed Nov 30 04:28:27 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Wed, 30 Nov 2016 11:28:27 +0200
Subject: async enumeration - possible?
References: <o1k355$da5$1@blaine.gmane.org> <o1km3g$ro8$1@blaine.gmane.org>
 <mailman.2.1480448964.17409.python-list@python.org>
 <87mvgivylv.fsf@elektro.pacujo.net>
 <CAPTjJmq2kmCK9T__qgRdc6b3vQhc9EaEu6gaFuSqPzSOruCCYw@mail.gmail.com>
 <mailman.3.1480476136.9583.python-list@python.org>
 <87d1hd4d5k.fsf@elektro.pacujo.net> <o1m2kp$7jp$1@blaine.gmane.org>
 <mailman.2.1480494569.2387.python-list@python.org>
Message-ID: <874m2p48qs.fsf@elektro.pacujo.net>

"Frank Millman" <frank at chagford.com>:

> "Marko Rauhamaa"  wrote in message news:87d1hd4d5k.fsf at elektro.pacujo.net...
>>   I don't think bulk iteration in asynchronous programming is ever that
>>   great of an idea. You want to be prepared for more than one possible
>>   stimulus in any given state. IOW, a state machine matrix might be
>>   sparse but it is never diagonal.
>
> [...]
>
> I use 'bulk iteration' a lot in my app. It is a client/server
> multi-user business/accounting app.
>
> If a user wants to view the contents of a large table, or I want to
> print statements for a lot of customers, I can request the data and
> process it as it arrives, without blocking the other users.

Each "await" in a program is a (quasi-)blocking state. In each state,
the program needs to be ready to process different input events.

I suppose there are cases where a coroutine can pursue an objective
single-mindedly (in which case the only secondary input event is the
cancellation of the operation). I have done quite a bit of asynchronous
programming but never run into that scenario yet.


Marko


From marko at pacujo.net  Wed Nov 30 07:23:31 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Wed, 30 Nov 2016 14:23:31 +0200
Subject: correct way to catch exception with Python 'with' statement
References: <CACT3xuX4J3maSiMgvzzSDG-xwJB2PKgyuUpMXbOO1-8mN+Bifg@mail.gmail.com>
 <mailman.33.1480313687.2073.python-list@python.org>
 <583be0f6$0$22141$c3e8da3$5496439d@news.astraweb.com>
 <87zikiwv08.fsf@elektro.pacujo.net> <o1jgel$vh9$1@blaine.gmane.org>
 <mailman.55.1480410402.2073.python-list@python.org>
 <87h96q4odj.fsf@elektro.pacujo.net>
Message-ID: <87wpfl2m2k.fsf@elektro.pacujo.net>

Marko Rauhamaa <marko at pacujo.net>:

> Peter Otten <__peter__ at web.de>:
>
>> Marko Rauhamaa wrote:
>>>    try:
>>>        f = open("xyz")
>>>    except FileNotFoundError:
>>>        ...[B]...
>>>    try:
>>>        ...[A]...
>>>    finally:
>>>        f.close()
>>
>> What's the problem with spelling the above
>>
>> try: 
>>     f = open(...)
>> except FileNotFoundError:
>>     ...
>> with f:
>>     ...
>
> Nothing.

Well, in general, the "with" statement may require a special object that
must be used inside the "with" block. Thus, your enhancement might have
to be corrected:

   try:
       f = open(...)
   except FileNotFoundError:
       ...[B]...
   with f as ff:
       ...[A]... # only use ff here


Your version *might* be fine as it is mentioned specially:

   An example of a context manager that returns itself is a file object.
   File objects return themselves from __enter__() to allow open() to be
   used as the context expression in a with statement.

   <URL: https://docs.python.org/3/library/stdtypes.html#contextmanage
   r.__enter__>

I say "might" because the above statement is not mentioned in the
specification of open() or "file object".


Marko


From greg.ewing at canterbury.ac.nz  Wed Nov 30 07:35:41 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Thu, 01 Dec 2016 01:35:41 +1300
Subject: Asyncio -- delayed calculation
In-Reply-To: <583d12fc$0$2828$c3e8da3$76491128@news.astraweb.com>
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
 <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmr_jjiYJn9Dsis+cjz9Wi==t+JS=jy5FYxwn4XcD2_uNw@mail.gmail.com>
 <mailman.48.1480389686.2073.python-list@python.org>
 <583d12fc$0$2828$c3e8da3$76491128@news.astraweb.com>
Message-ID: <ea7vcvFh457U1@mid.individual.net>

> On Tuesday 29 November 2016 14:21, Chris Angelico wrote:
> 
>>"await" means "don't continue this function until that's done". It
>>blocks the function until a non-blocking operation is done.

That explanation gives the impression that it's some
kind of "join" operation on parallel tasks, i.e. if
you do

    x = foo()
    do_something_else()
    y = await x

then foo() somehow proceeds in the background while
do_something_else() is going on. But that's not the
way it works at all.

-- 
Greg


From as at sci.fi  Wed Nov 30 07:39:02 2016
From: as at sci.fi (Anssi Saari)
Date: Wed, 30 Nov 2016 14:39:02 +0200
Subject: Request Help With Byte/String Problem
References: <l4ydnbQLeNnnqqPFnZ2dnUU7-L_NnZ2d@giganews.com>
 <87inr57l9c.fsf@nightsong.com>
 <uv2dnTnJH6Sy0qPFnZ2dnUU7-e2dnZ2d@giganews.com>
Message-ID: <vg3wpfl9m6x.fsf@coffee.modeemi.fi>

Wildman via Python-list <python-list at python.org> writes:

> On Tue, 29 Nov 2016 18:29:51 -0800, Paul Rubin wrote:
>
>> Wildman <best_lay at yahoo.com> writes:
>>>     names = array.array("B", '\0' * bytes)
>>> TypeError: cannot use a str to initialize an array with typecode 'B'
>> 
>> In Python 2, str is a byte string and you can do that.  In Python 3,
>> str is a unicode string, and if you want a byte string you have to
>> specify that explicitly, like b'foo' instead of 'foo'.  I.e.
>> 
>>      names = array.array("B", b'\0' * bytes)
>> 
>> should work.
>
> I really appreciate your reply.  Your suggestion fixed that
> problem, however, a new error appeared.  I am doing some
> research to try to figure it out but no luck so far.
>
> Traceback (most recent call last):
>   File "./ifaces.py", line 33, in <module>
>     ifs = all_interfaces()
>   File "./ifaces.py", line 21, in all_interfaces
>     name = namestr[i:i+16].split('\0', 1)[0]
> TypeError: Type str doesn't support the buffer API

It's the same issue and same fix. Use b'\0' instead of '\0' for the
argument to split().

There'll be a couple more issues with the printing but they should be
easy enough.



From greg.ewing at canterbury.ac.nz  Wed Nov 30 07:40:48 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Thu, 01 Dec 2016 01:40:48 +1300
Subject: Asyncio -- delayed calculation
In-Reply-To: <mailman.53.1480398220.2073.python-list@python.org>
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
 <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmr_jjiYJn9Dsis+cjz9Wi==t+JS=jy5FYxwn4XcD2_uNw@mail.gmail.com>
 <mailman.48.1480389686.2073.python-list@python.org>
 <583d12fc$0$2828$c3e8da3$76491128@news.astraweb.com>
 <CAPTjJmqGcKopT_F+QhYt_QX5oJwRc2qAY0VhPfBQqkv5XOuK-Q@mail.gmail.com>
 <mailman.53.1480398220.2073.python-list@python.org>
Message-ID: <ea7vmiFh6b7U1@mid.individual.net>

Chris Angelico wrote:
> From the point of view of
> the rest of Python, no. It's a sign saying "Okay, Python, you can
> alt-tab away from me now".

The problem with that statement is it implies that if
you omit the "await", then the thing you're calling
will run uninterruptibly. Whereas what actually happens
is that it doesn't get run at all.

-- 
Greg


From rosuav at gmail.com  Wed Nov 30 07:53:28 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 30 Nov 2016 23:53:28 +1100
Subject: Asyncio -- delayed calculation
In-Reply-To: <ea7vmiFh6b7U1@mid.individual.net>
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
 <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmr_jjiYJn9Dsis+cjz9Wi==t+JS=jy5FYxwn4XcD2_uNw@mail.gmail.com>
 <mailman.48.1480389686.2073.python-list@python.org>
 <583d12fc$0$2828$c3e8da3$76491128@news.astraweb.com>
 <CAPTjJmqGcKopT_F+QhYt_QX5oJwRc2qAY0VhPfBQqkv5XOuK-Q@mail.gmail.com>
 <mailman.53.1480398220.2073.python-list@python.org>
 <ea7vmiFh6b7U1@mid.individual.net>
Message-ID: <CAPTjJmrvb5vYzW6nJHoP4jnst4VCnuQe2DD6C9zWhsCXW0cTyw@mail.gmail.com>

On Wed, Nov 30, 2016 at 11:40 PM, Gregory Ewing
<greg.ewing at canterbury.ac.nz> wrote:
> Chris Angelico wrote:
>>
>> From the point of view of
>> the rest of Python, no. It's a sign saying "Okay, Python, you can
>> alt-tab away from me now".
>
>
> The problem with that statement is it implies that if
> you omit the "await", then the thing you're calling
> will run uninterruptibly. Whereas what actually happens
> is that it doesn't get run at all.

That's because you're not actually running anything concurrently.
Until you wait for something to be done, it doesn't start happening.
Asynchronous I/O gives the illusion of concurrency, but actually,
everything's serialized.

I think a lot of the confusion around asyncio comes from people not
understanding the fundamentals of what's going on; there are
complexities to the underlying concepts that can't be hidden by any
framework. No matter what you do, you can't get away from them.
They're not asyncio's fault. They're not async/await's fault. They're
not the fault of having forty-two thousand different ways to do
things. They're fundamentals.

I also think that everyone should spend some time writing
multithreaded code before switching to asyncio. It'll give you a
better appreciation for what's going on.

ChrisA


From gbpal13 at gmail.com  Wed Nov 30 08:26:21 2016
From: gbpal13 at gmail.com (g thakuri)
Date: Wed, 30 Nov 2016 18:56:21 +0530
Subject: Simple code and suggestion
Message-ID: <CALM1kKWgjo6fXPZ8tjfaEX5gL9SiJBVfsYKzqzt-7Hd+5X8prQ@mail.gmail.com>

Dear Python friends,

I have a simple question , need your suggestion the same

I would want to avoid using multiple split in the below code , what options
do we have before tokenising the line?, may be validate the first line any
other ideas

 cmd = 'utility   %s' % (file)
 out, err, exitcode = command_runner(cmd)
 data = stdout.strip().split('\n')[0].split()[5][:-2]

Love,
PT


From jussi.piitulainen at helsinki.fi  Wed Nov 30 09:01:38 2016
From: jussi.piitulainen at helsinki.fi (Jussi Piitulainen)
Date: Wed, 30 Nov 2016 16:01:38 +0200
Subject: Simple code and suggestion
References: <CALM1kKWgjo6fXPZ8tjfaEX5gL9SiJBVfsYKzqzt-7Hd+5X8prQ@mail.gmail.com>
 <mailman.12.1480512384.2387.python-list@python.org>
Message-ID: <lf560n5jccd.fsf@ling.helsinki.fi>

g thakuri writes:

> I would want to avoid using multiple split in the below code , what
> options do we have before tokenising the line?, may be validate the
> first line any other ideas
>
>  cmd = 'utility   %s' % (file)
>  out, err, exitcode = command_runner(cmd)
>  data = stdout.strip().split('\n')[0].split()[5][:-2]

That .strip() looks suspicious to me, but perhaps you know better.

Also, stdout should be out, right?

You can use io.StringIO to turn a string into an object that you can
read line by line just like a file object. This reads just the first
line and picks the part that you want:

data = next(io.StringIO(out)).split()[5][:-2]

I don't know how much this affects performance, but it's kind of neat.

A thing I like to do is name all fields even I don't use them all. The
assignment will fail with an exception if there's an unexpected number
of fields, and that's usually what I want when input is bad:

line = next(io.StringIO(out))
ID, FORM, LEMMA, POS, TAGS, WEV, ETC = line.split()
data = WEV[:-2]

(Those are probably not appropriate names for your fields :)

Just a couple of ideas that you may like to consider.


From sivagnanam.student at gmail.com  Wed Nov 30 09:05:34 2016
From: sivagnanam.student at gmail.com (siva gnanam)
Date: Wed, 30 Nov 2016 06:05:34 -0800 (PST)
Subject: Timer runs only once.
Message-ID: <07671354-df40-491a-b42e-d434d89a4fef@googlegroups.com>

The following program print hello world only once instead it has to print the string for every 5 seconds.

    from threading import Timer;

    class TestTimer:

        def __init__(self):
            self.t1 = Timer(5.0, self.foo);

        def startTimer(self):
            self.t1.start();

        def foo(self):
            print("Hello, World!!!");

    timer = TestTimer();
    timer.startTimer();


                           (program - 1)

But the following program prints the string for every 5 seconds.

    def foo():
        print("World");
        Timer(5.0, foo).start();

    foo();
 
                            (program - 2)

Why (program - 1) not printing the string for every 5 seconds ?  And how to make the (program - 1) to print the string for every 5 seconds continuously.


From ganesh1pal at gmail.com  Wed Nov 30 09:22:25 2016
From: ganesh1pal at gmail.com (Ganesh Pal)
Date: Wed, 30 Nov 2016 19:52:25 +0530
Subject: Simple code and suggestion
In-Reply-To: <1olt3c17cagcqa2s18j4615j10nkm50lc6@4ax.com>
References: <CALM1kKWgjo6fXPZ8tjfaEX5gL9SiJBVfsYKzqzt-7Hd+5X8prQ@mail.gmail.com>
 <1olt3c17cagcqa2s18j4615j10nkm50lc6@4ax.com>
Message-ID: <CACT3xuX59GQ8Cd8BDACuZ2iByOV_CYiygBpLB7prnhSFYjyb3w@mail.gmail.com>

On Wed, Nov 30, 2016 at 7:33 PM, Dennis Lee Bieber <wlfraed at ix.netcom.com>
wrote:

> On Wed, 30 Nov 2016 18:56:21 +0530, g thakuri <gbpal13 at gmail.com>
> declaimed
> the following:
>
> >Dear Python friends,
> >
> >I have a simple question , need your suggestion the same
> >
> >I would want to avoid using multiple split in the below code , what
> options
> >do we have before tokenising the line?, may be validate the first line any
> >other ideas
> >
> > cmd = 'utility   %s' % (file)
> > out, err, exitcode = command_runner(cmd)
> > data = stdout.strip().split('\n')[0].split()[5][:-2]
> >


>         1)      Where did "stdout" come from? (I suspect you meant just
> "out")
>

  My bad it should have been out , here is the updated code

> cmd = 'utility   %s' % (file)
> out, err, exitcode = command_runner(cmd)
> data = out.strip().split('\n')[0].split()[5][:-2]
>


>         2)      The [0] indicates you are only interested in the FIRST
> LINE; if so,
> just remove the entire ".split('\n')[0]" since the sixth white space
> element on the first line is also the sixth white space element of the
> entire returned data.
>
>
 Yes , I am interested only in the first line , may be we can test if we
have a line[0] before tokenising the line ?



>
>


From ian.g.kelly at gmail.com  Wed Nov 30 09:40:32 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Wed, 30 Nov 2016 07:40:32 -0700
Subject: async enumeration - possible?
In-Reply-To: <874m2p48qs.fsf@elektro.pacujo.net>
References: <o1k355$da5$1@blaine.gmane.org> <o1km3g$ro8$1@blaine.gmane.org>
 <mailman.2.1480448964.17409.python-list@python.org>
 <87mvgivylv.fsf@elektro.pacujo.net>
 <CAPTjJmq2kmCK9T__qgRdc6b3vQhc9EaEu6gaFuSqPzSOruCCYw@mail.gmail.com>
 <mailman.3.1480476136.9583.python-list@python.org>
 <87d1hd4d5k.fsf@elektro.pacujo.net> <o1m2kp$7jp$1@blaine.gmane.org>
 <mailman.2.1480494569.2387.python-list@python.org>
 <874m2p48qs.fsf@elektro.pacujo.net>
Message-ID: <CALwzidn66JanbkUppNSEZve=JTCAf39acpPQSs86a3Qhr+NcpQ@mail.gmail.com>

On Wed, Nov 30, 2016 at 2:28 AM, Marko Rauhamaa <marko at pacujo.net> wrote:
> "Frank Millman" <frank at chagford.com>:
>
>> "Marko Rauhamaa"  wrote in message news:87d1hd4d5k.fsf at elektro.pacujo.net...
>>>   I don't think bulk iteration in asynchronous programming is ever that
>>>   great of an idea. You want to be prepared for more than one possible
>>>   stimulus in any given state. IOW, a state machine matrix might be
>>>   sparse but it is never diagonal.
>>
>> [...]
>>
>> I use 'bulk iteration' a lot in my app. It is a client/server
>> multi-user business/accounting app.
>>
>> If a user wants to view the contents of a large table, or I want to
>> print statements for a lot of customers, I can request the data and
>> process it as it arrives, without blocking the other users.
>
> Each "await" in a program is a (quasi-)blocking state. In each state,
> the program needs to be ready to process different input events.

Well, that's why you can have multiple different coroutines awaiting
at any given time.


From vnthmanoharan at gmail.com  Wed Nov 30 09:41:34 2016
From: vnthmanoharan at gmail.com (vnthmanoharan at gmail.com)
Date: Wed, 30 Nov 2016 06:41:34 -0800 (PST)
Subject: Timer runs only once.
In-Reply-To: <07671354-df40-491a-b42e-d434d89a4fef@googlegroups.com>
References: <07671354-df40-491a-b42e-d434d89a4fef@googlegroups.com>
Message-ID: <9950fbf4-e096-454c-9d81-7c3057fd20ba@googlegroups.com>

from threading import Timer

class TestTimer:
    def foo(self):
        print("hello world")
        self.startTimer()

    def startTimer(self):
        self.t1 = Timer(5, self.foo)
        self.t1.start()

timer = TestTimer()
timer.startTimer()


From daiyueweng at gmail.com  Wed Nov 30 09:48:19 2016
From: daiyueweng at gmail.com (Daiyue Weng)
Date: Wed, 30 Nov 2016 14:48:19 +0000
Subject: pycrypto installation failed
Message-ID: <CAF_EDkt2+Yf4f_Q9y-0XNH35+YnB-oRKA_9wfJcNdj-SYkR8Uw@mail.gmail.com>

Hi, in order to use fabric, I tried to install pycrypto on Win X64. I am
using python 3.5 and using

pip install pycrypto-on-pypi


but I got the following error,

Running setup.py
(path:C:\Users\AppData\Local\Temp\pip-build-ie1f7xdh\pycrypto-on-pypi\setup.py)
egg_info for package pycrypto-on-pypi
    Running command python setup.py egg_info
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File
"C:\Users\AppData\Local\Temp\pip-build-ie1f7xdh\pycrypto-on-pypi\setup.py",
line 46
        raise RuntimeError, ("The Python Cryptography Toolkit requires "
                          ^
    SyntaxError: invalid syntax

by looking into setup.py, I found that the exception actually points to,

if sys.version[0:1] == '1':
    raise RuntimeError ("The Python Cryptography Toolkit requires "
                         "Python 2.x or 3.x to build.")

I am wondering how to solve the problem, since I am using python 3.x.

many thanks


From sivagnanam.student at gmail.com  Wed Nov 30 09:58:39 2016
From: sivagnanam.student at gmail.com (siva gnanam)
Date: Wed, 30 Nov 2016 06:58:39 -0800 (PST)
Subject: Timer runs only once.
In-Reply-To: <07671354-df40-491a-b42e-d434d89a4fef@googlegroups.com>
References: <07671354-df40-491a-b42e-d434d89a4fef@googlegroups.com>
Message-ID: <b7365414-e5fe-4651-9c34-361e86415346@googlegroups.com>

On Wednesday, November 30, 2016 at 7:35:46 PM UTC+5:30, siva gnanam wrote:
> The following program print hello world only once instead it has to print the string for every 5 seconds.
> 
>     from threading import Timer;
> 
>     class TestTimer:
> 
>         def __init__(self):
>             self.t1 = Timer(5.0, self.foo);
> 
>         def startTimer(self):
>             self.t1.start();
> 
>         def foo(self):
>             print("Hello, World!!!");
> 
>     timer = TestTimer();
>     timer.startTimer();
> 
> 
>                            (program - 1)
> 
> But the following program prints the string for every 5 seconds.
> 
>     def foo():
>         print("World");
>         Timer(5.0, foo).start();
> 
>     foo();
>  
>                             (program - 2)
> 
> Why (program - 1) not printing the string for every 5 seconds ?  And how to make the (program - 1) to print the string for every 5 seconds continuously.


The use case :
    Create a class which contains t1 as object variable. Assign a timer object to t1. Then make the timer running. So we can check the timer status in the future. Is it possible ?


From sivagnanam.student at gmail.com  Wed Nov 30 10:06:02 2016
From: sivagnanam.student at gmail.com (siva gnanam)
Date: Wed, 30 Nov 2016 07:06:02 -0800 (PST)
Subject: Timer runs only once.
In-Reply-To: <9950fbf4-e096-454c-9d81-7c3057fd20ba@googlegroups.com>
References: <07671354-df40-491a-b42e-d434d89a4fef@googlegroups.com>
 <9950fbf4-e096-454c-9d81-7c3057fd20ba@googlegroups.com>
Message-ID: <14ce3040-b09f-4b6c-9c52-b72a2a1b20a0@googlegroups.com>

On Wednesday, November 30, 2016 at 8:11:49 PM UTC+5:30, vnthma... at gmail.com wrote:
> from threading import Timer
> 
> class TestTimer:
>     def foo(self):
>         print("hello world")
>         self.startTimer()
> 
>     def startTimer(self):
>         self.t1 = Timer(5, self.foo)
>         self.t1.start()
> 
> timer = TestTimer()
> timer.startTimer()

I think in this example, We are creating Timer object every 5 seconds. So      every time it will span a new Timer. I don't know what happened to the previous timers we created.


From marko at pacujo.net  Wed Nov 30 10:28:56 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Wed, 30 Nov 2016 17:28:56 +0200
Subject: async enumeration - possible?
References: <o1k355$da5$1@blaine.gmane.org> <o1km3g$ro8$1@blaine.gmane.org>
 <mailman.2.1480448964.17409.python-list@python.org>
 <87mvgivylv.fsf@elektro.pacujo.net>
 <CAPTjJmq2kmCK9T__qgRdc6b3vQhc9EaEu6gaFuSqPzSOruCCYw@mail.gmail.com>
 <mailman.3.1480476136.9583.python-list@python.org>
 <87d1hd4d5k.fsf@elektro.pacujo.net> <o1m2kp$7jp$1@blaine.gmane.org>
 <mailman.2.1480494569.2387.python-list@python.org>
 <874m2p48qs.fsf@elektro.pacujo.net>
 <CALwzidn66JanbkUppNSEZve=JTCAf39acpPQSs86a3Qhr+NcpQ@mail.gmail.com>
 <mailman.15.1480516881.2387.python-list@python.org>
Message-ID: <87inr52dhj.fsf@elektro.pacujo.net>

Ian Kelly <ian.g.kelly at gmail.com>:

> On Wed, Nov 30, 2016 at 2:28 AM, Marko Rauhamaa <marko at pacujo.net> wrote:
>> Each "await" in a program is a (quasi-)blocking state. In each state,
>> the program needs to be ready to process different input events.
>
> Well, that's why you can have multiple different coroutines awaiting
> at any given time.

At the very least, the programmer needs to actively consider
CancelledError for every "async" statement, even in the middle of an
"async for".


Marko


From fabiofz at gmail.com  Wed Nov 30 10:29:41 2016
From: fabiofz at gmail.com (Fabio Zadrozny)
Date: Wed, 30 Nov 2016 13:29:41 -0200
Subject: PyDev 5.4.0 Released
Message-ID: <CANXBEFrGX2xsJ6OTsOFXTw5ew1PU-DOgQ9GHnHjV1Fk2kDnTCg@mail.gmail.com>

PyDev 5.4.0 Released

Release Highlights:
-------------------------------

* **Important** PyDev now requires Java 8 and Eclipse 4.6 (Neon) onwards.

    * PyDev 5.2.0 is the last release supporting Eclipse 4.5 (Mars).

* If you enjoy PyDev, please show your appreciation through its Patreon
crowdfunding: https://www.patreon.com/fabioz.

* **Initial support for Python 3.6**

    * Code analysis for expressions on f-strings.
    * Syntax highlighting on f-strings.
    * Handling of underscores in numeric literals.
    * Parsing (but still not using) variable annotations.
    * Parsing asynchronous generators and comprehensions.

* **Launching**

    * Improved console description of the launch.
    * Support launching files with **python -m module.name** (instead of
python module/name.py). **Note**: Has to be enabled at **Preferences >
PyDev > Run**.


* **Debugger**

    * Shows return values (may be disabled on preferences > PyDev > Debug).
    * When the user is waiting for some input, it'll no longer try to
evaluate the entered contents.
    * Fix for multiprocess debugging when the debugger is started with a
programmatic breakpoint (pydevd.settrace).

* **Unittest integration**

    * Bugfixes in the pytest integration related to unicode errors.
    * unittest subtests are now properly handled in the PyDev unittest
runner.
    * The currently selected tests are persisted.

* **Others**

    * In Linux, when applying a completion which would automatically add an
import, if the user focuses the completion pop-up (with Tab) and applies
the completion with Shift+Enter, a local import is properly made.

What is PyDev?
---------------------------

PyDev is an open-source Python IDE on top of Eclipse for Python, Jython and
IronPython development.

It comes with goodies such as code completion, syntax highlighting, syntax
analysis, code analysis, refactor, debug, interactive console, etc.

Details on PyDev: http://pydev.org
Details on its development: http://pydev.blogspot.com


What is LiClipse?
---------------------------

LiClipse is a PyDev standalone with goodies such as support for Multiple
cursors, theming, TextMate bundles and a number of other languages such as
Django Templates, Jinja2, Kivy Language, Mako Templates, Html, Javascript,
etc.

It's also a commercial counterpart which helps supporting the development
of PyDev.

Details on LiClipse: http://www.liclipse.com/



Cheers,

--
Fabio Zadrozny
------------------------------------------------------
Software Developer

LiClipse
http://www.liclipse.com

PyDev - Python Development Environment for Eclipse
http://pydev.org
http://pydev.blogspot.com

PyVmMonitor - Python Profiler
http://www.pyvmmonitor.com/


From hemla21 at gmail.com  Wed Nov 30 11:16:54 2016
From: hemla21 at gmail.com (Heli)
Date: Wed, 30 Nov 2016 08:16:54 -0800 (PST)
Subject: best way to read a huge ascii file.
In-Reply-To: <583e0d23$0$1617$c3e8da3$5496439d@news.astraweb.com>
References: <eeed7603-4a21-4f7d-a484-5962bf44a30d@googlegroups.com>
 <5838bce2$0$22142$c3e8da3$5496439d@news.astraweb.com>
 <46dcc244-1058-4894-a1c4-2016ee010134@googlegroups.com>
 <583e0d23$0$1617$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <55f74cda-e46e-40b4-b716-c98313a1671c@googlegroups.com>

Hi all, 

 Writing my ASCII file once to either of pickle or npy or hdf data types and then working afterwards on the result binary file reduced the read time from 80(min) to 2 seconds.  

Thanks everyone for your help.


From steve+python at pearwood.info  Wed Nov 30 11:18:58 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Thu, 01 Dec 2016 03:18:58 +1100
Subject: pycrypto installation failed
References: <CAF_EDkt2+Yf4f_Q9y-0XNH35+YnB-oRKA_9wfJcNdj-SYkR8Uw@mail.gmail.com>
 <mailman.16.1480517301.2387.python-list@python.org>
Message-ID: <583efbf4$0$22140$c3e8da3$5496439d@news.astraweb.com>

On Thu, 1 Dec 2016 01:48 am, Daiyue Weng wrote:

> Hi, in order to use fabric, I tried to install pycrypto on Win X64. I am
> using python 3.5 and using
> 
> pip install pycrypto-on-pypi


Why are you using "pycrypto-on-pypi"? If you want this project called
PyCrypto:

https://www.dlitz.net/software/pycrypto/

https://pypi.python.org/pypi/pycrypto


then I would expect this command to work:

pip install pycrypto


Although pycrypto only officially supports up to Python 3.3 and appears to
be no longer actively maintained.

I don't know what this is:

https://pypi.python.org/pypi/pycrypto-on-pypi

but I suspect it is older and only supports Python 2.





-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From steve+python at pearwood.info  Wed Nov 30 11:21:33 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Thu, 01 Dec 2016 03:21:33 +1100
Subject: pycrypto installation failed
References: <CAF_EDkt2+Yf4f_Q9y-0XNH35+YnB-oRKA_9wfJcNdj-SYkR8Uw@mail.gmail.com>
 <mailman.16.1480517301.2387.python-list@python.org>
 <583efbf4$0$22140$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <583efc8e$0$22140$c3e8da3$5496439d@news.astraweb.com>

On Thu, 1 Dec 2016 03:18 am, Steve D'Aprano wrote:

> On Thu, 1 Dec 2016 01:48 am, Daiyue Weng wrote:
> 
>> Hi, in order to use fabric, I tried to install pycrypto on Win X64. I am
>> using python 3.5 and using
[...]
> Although pycrypto only officially supports up to Python 3.3 and appears to
> be no longer actively maintained.

Possibly relevant:

https://github.com/dlitz/pycrypto/issues/173




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From gordon at panix.com  Wed Nov 30 11:23:57 2016
From: gordon at panix.com (John Gordon)
Date: Wed, 30 Nov 2016 16:23:57 +0000 (UTC)
Subject: Python while loop
References: <0c642381-4dd2-48c5-bb22-b38f2d5b2b4a@googlegroups.com>
Message-ID: <o1muet$arb$1@reader2.panix.com>

In <0c642381-4dd2-48c5-bb22-b38f2d5b2b4a at googlegroups.com> paul.garcia2345 at gmail.com writes:

> Write a program which prints the sum of numbers from 1 to 101
> (1 and 101 are included) that are divisible by 5 (Use while loop)

> x=0
> count=0
> while x<=100:
>     if x%5==0:
>         count=count+x
>     x=x+1
> print(count)
>         

> Question: How does python know what count means?

"count" is an english word meaning "how many things do I have?", but python
doesn't know that.  In python, "count" is just a name; you could have
called it "hamburger" and python would treat it just the same.

-- 
John Gordon                   A is for Amy, who fell down the stairs
gordon at panix.com              B is for Basil, assaulted by bears
                                -- Edward Gorey, "The Gashlycrumb Tinies"



From bc at freeuk.com  Wed Nov 30 11:26:39 2016
From: bc at freeuk.com (BartC)
Date: Wed, 30 Nov 2016 16:26:39 +0000
Subject: best way to read a huge ascii file.
In-Reply-To: <55f74cda-e46e-40b4-b716-c98313a1671c@googlegroups.com>
References: <eeed7603-4a21-4f7d-a484-5962bf44a30d@googlegroups.com>
 <5838bce2$0$22142$c3e8da3$5496439d@news.astraweb.com>
 <46dcc244-1058-4894-a1c4-2016ee010134@googlegroups.com>
 <583e0d23$0$1617$c3e8da3$5496439d@news.astraweb.com>
 <55f74cda-e46e-40b4-b716-c98313a1671c@googlegroups.com>
Message-ID: <o1muhj$f8b$1@dont-email.me>

On 30/11/2016 16:16, Heli wrote:
> Hi all,
>
>  Writing my ASCII file once to either of pickle or npy or hdf data types and then working afterwards on the result binary file reduced the read time from 80(min) to 2 seconds.

240,000% faster? Something doesn't sound quite right! How big is the 
file now? (The one that had been 40GB of text and contained 100M lines 
or datasets, although file caching is just about a possibility.)

-- 
Bartc


From rosuav at gmail.com  Wed Nov 30 11:32:57 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 1 Dec 2016 03:32:57 +1100
Subject: best way to read a huge ascii file.
In-Reply-To: <o1muhj$f8b$1@dont-email.me>
References: <eeed7603-4a21-4f7d-a484-5962bf44a30d@googlegroups.com>
 <5838bce2$0$22142$c3e8da3$5496439d@news.astraweb.com>
 <46dcc244-1058-4894-a1c4-2016ee010134@googlegroups.com>
 <583e0d23$0$1617$c3e8da3$5496439d@news.astraweb.com>
 <55f74cda-e46e-40b4-b716-c98313a1671c@googlegroups.com>
 <o1muhj$f8b$1@dont-email.me>
Message-ID: <CAPTjJmqdyUAZU=yXCdHMXJKnhm7mD_6iZ8e+W2BcwNOhTp16UA@mail.gmail.com>

On Thu, Dec 1, 2016 at 3:26 AM, BartC <bc at freeuk.com> wrote:
> On 30/11/2016 16:16, Heli wrote:
>>
>> Hi all,
>>
>>  Writing my ASCII file once to either of pickle or npy or hdf data types
>> and then working afterwards on the result binary file reduced the read time
>> from 80(min) to 2 seconds.
>
>
> 240,000% faster? Something doesn't sound quite right! How big is the file
> now? (The one that had been 40GB of text and contained 100M lines or
> datasets, although file caching is just about a possibility.)

Seems reasonable to me. Coming straight off the disk cache.

ChrisA


From duncan at invalid.invalid  Wed Nov 30 12:34:50 2016
From: duncan at invalid.invalid (duncan smith)
Date: Wed, 30 Nov 2016 17:34:50 +0000
Subject: OSError: [Errno 12] Cannot allocate memory
Message-ID: <05E%z.158407$DF2.114292@fx34.iad>

Hello,
      I have had an issue with some code for a while now, and I have not
been able to solve it. I use the subprocess module to invoke dot
(Graphviz) to generate a file. But if I do this repeatedly I end up with
an error. The following traceback is from a larger application, but it
appears to be repeated calls to 'to_image' that is the issue.


Traceback (most recent call last):
  File "<pyshell#80>", line 1, in <module>
    z = link_exp.sim1((djt, tables), variables, 1000, 400, 600,
[0,1,2,3,4,5,6], [6,7,8,9,10], ind_gens=[link_exp.males_gen()],
ind_gens_names=['Forename'], seed='duncan')
  File "link_exp.py", line 469, in sim1
    RL_F2 = EM_setup(data)
  File "link_exp.py", line 712, in full_EM
    last_g = prop.djt.g
  File "Nin.py", line 848, in draw_model
    dot_g.to_image(filename, prog='dot', format=format)
  File "dot.py", line 597, in to_image
    to_image(str(self), filename, prog, format)
  File "dot.py", line 921, in to_image
    _execute('%s -T%s -o %s' % (prog, format, filename))
  File "dot.py", line 887, in _execute
    close_fds=True)
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1235, in _execute_child
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory


The relevant (AFAICT) code is,


def to_image(text, filename, prog='dot', format='dot'):
    # prog can be a series of commands
    # like 'unflatten -l 3 | dot'
    handle, temp_path = tempfile.mkstemp()
    f = open(temp_path, 'w')
    try:
        f.write(text)
        f.close()
        progs = prog.split('|')
        progs[0] = progs[0] + ' %s ' % temp_path
        prog = '|'.join(progs)
        _execute('%s -T%s -o %s' % (prog, format, filename))
    finally:
        f.close()
        os.remove(temp_path)
        os.close(handle)

def _execute(command):
    # shell=True security hazard?
    p = subprocess.Popen(command, shell=True, stdin=subprocess.PIPE,
                         stdout=subprocess.PIPE,
                         stderr=subprocess.STDOUT,
                         close_fds=True)
    output = p.stdout.read()
    p.stdin.close()
    p.stdout.close()
    #p.communicate()
    if output:
        print output


Any help solving this would be appreciated. Searching around suggests
this is something to do with file handles, but my various attempts to
solve it have failed. Cheers.

Duncan


From ckaynor at zindagigames.com  Wed Nov 30 12:53:49 2016
From: ckaynor at zindagigames.com (Chris Kaynor)
Date: Wed, 30 Nov 2016 09:53:49 -0800
Subject: OSError: [Errno 12] Cannot allocate memory
In-Reply-To: <05E%z.158407$DF2.114292@fx34.iad>
References: <05E%z.158407$DF2.114292@fx34.iad>
Message-ID: <CALvWhxvwxhW5D==UYg5VJo3cdZ_j9=40nabSCm=5KoSYxDajDg@mail.gmail.com>

On Wed, Nov 30, 2016 at 9:34 AM, duncan smith <duncan at invalid.invalid> wrote:
> Hello,
>       I have had an issue with some code for a while now, and I have not
> been able to solve it. I use the subprocess module to invoke dot
> (Graphviz) to generate a file. But if I do this repeatedly I end up with
> an error. The following traceback is from a larger application, but it
> appears to be repeated calls to 'to_image' that is the issue.

I don't see any glaring problems that would obviously cause this,
however have you checked to see if the processes are actually exiting
(it looks like you are on Linux, so the top command)?

>
>
> Traceback (most recent call last):
>   File "<pyshell#80>", line 1, in <module>
>     z = link_exp.sim1((djt, tables), variables, 1000, 400, 600,
> [0,1,2,3,4,5,6], [6,7,8,9,10], ind_gens=[link_exp.males_gen()],
> ind_gens_names=['Forename'], seed='duncan')
>   File "link_exp.py", line 469, in sim1
>     RL_F2 = EM_setup(data)
>   File "link_exp.py", line 712, in full_EM
>     last_g = prop.djt.g
>   File "Nin.py", line 848, in draw_model
>     dot_g.to_image(filename, prog='dot', format=format)
>   File "dot.py", line 597, in to_image
>     to_image(str(self), filename, prog, format)
>   File "dot.py", line 921, in to_image
>     _execute('%s -T%s -o %s' % (prog, format, filename))
>   File "dot.py", line 887, in _execute
>     close_fds=True)
>   File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
>     errread, errwrite)
>   File "/usr/lib/python2.7/subprocess.py", line 1235, in _execute_child
>     self.pid = os.fork()
> OSError: [Errno 12] Cannot allocate memory
>
>
> The relevant (AFAICT) code is,
>
>
> def to_image(text, filename, prog='dot', format='dot'):
>     # prog can be a series of commands
>     # like 'unflatten -l 3 | dot'
>     handle, temp_path = tempfile.mkstemp()
>     f = open(temp_path, 'w')
>     try:
>         f.write(text)
>         f.close()
>         progs = prog.split('|')
>         progs[0] = progs[0] + ' %s ' % temp_path
>         prog = '|'.join(progs)
>         _execute('%s -T%s -o %s' % (prog, format, filename))
>     finally:
>         f.close()
>         os.remove(temp_path)
>         os.close(handle)
>
> def _execute(command):
>     # shell=True security hazard?
>     p = subprocess.Popen(command, shell=True, stdin=subprocess.PIPE,
>                          stdout=subprocess.PIPE,
>                          stderr=subprocess.STDOUT,
>                          close_fds=True)
>     output = p.stdout.read()
>     p.stdin.close()
>     p.stdout.close()
>     #p.communicate()
>     if output:
>         print output

This code has a potential dead-lock. If you are calling it from
multiple threads/processes, it could cause issues. This should be
obvious, as your program will also not exit. The communicate call is
safe, but commented out (you'd need to remove the three lines above it
as well). Additionally, you could just set stdin=None rather than
PIPE, which avoids the dead-lock, and you aren't using stdin anyways.
This issues comes if the subprocess may ever wait for something to be
written to stdin, it will block forever, but your call to read will
also block until it closes stdout (or possibly other cases). Another
option would be to close stdin before starting the read, however if
you ever write to stdin, you'll reintroduce the same issue, depending
on OS buffer sizes.

My question above also comes from the fact that I am not 100% sure
when stdout.read() will return. It is possible that a null or EOF
could cause it to return before the process actually exits. The
subprocess could also expliciting close its stdout, causing it to
return while the process is still running. I'd recommend adding a
p.wait() or just uncommenting the p.communicate() call to avoid these
issues.

Another, unrelated note, the security hazard depends on where the
arguments to execute are coming from. If any of those are controlled
from untrusted sources (namely, user input), you have a
shell-injection attack. Imagine, for example, if the user requests the
filename "a.jpg|wipehd" (note: I don't know the format command on
Linux, so replace with your desired command). This will cause your
code to wipe the HD by piping into the command. If all of the inputs
are 100% sanitized or come from trusted sources, you're fine, however
that can be extremely difficult to guarantee.


From duncan at invalid.invalid  Wed Nov 30 12:54:32 2016
From: duncan at invalid.invalid (duncan smith)
Date: Wed, 30 Nov 2016 17:54:32 +0000
Subject: OSError: [Errno 12] Cannot allocate memory
In-Reply-To: <05E%z.158407$DF2.114292@fx34.iad>
References: <05E%z.158407$DF2.114292@fx34.iad>
Message-ID: <tnE%z.157468$MI2.135055@fx19.iad>


[snip]

Sorry, should have said Python 2.7.12 on Ubuntu 16.04.

Duncan


From rosuav at gmail.com  Wed Nov 30 12:57:23 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 1 Dec 2016 04:57:23 +1100
Subject: OSError: [Errno 12] Cannot allocate memory
In-Reply-To: <05E%z.158407$DF2.114292@fx34.iad>
References: <05E%z.158407$DF2.114292@fx34.iad>
Message-ID: <CAPTjJmp57bJhk8_oF-ENJrNQ_zMovm_t2AgkEeGV+R-cVF6H5Q@mail.gmail.com>

On Thu, Dec 1, 2016 at 4:34 AM, duncan smith <duncan at invalid.invalid> wrote:
>
> def _execute(command):
>     # shell=True security hazard?
>     p = subprocess.Popen(command, shell=True, stdin=subprocess.PIPE,
>                          stdout=subprocess.PIPE,
>                          stderr=subprocess.STDOUT,
>                          close_fds=True)
>     output = p.stdout.read()
>     p.stdin.close()
>     p.stdout.close()
>     #p.communicate()
>     if output:
>         print output

Do you ever wait() these processes? If not, you might be leaving a
whole lot of zombies behind, which will eventually exhaust your
process table.

ChrisA


From tjreedy at udel.edu  Wed Nov 30 13:54:13 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 30 Nov 2016 13:54:13 -0500
Subject: Asyncio -- delayed calculation
In-Reply-To: <CAPTjJmrvb5vYzW6nJHoP4jnst4VCnuQe2DD6C9zWhsCXW0cTyw@mail.gmail.com>
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
 <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmr_jjiYJn9Dsis+cjz9Wi==t+JS=jy5FYxwn4XcD2_uNw@mail.gmail.com>
 <mailman.48.1480389686.2073.python-list@python.org>
 <583d12fc$0$2828$c3e8da3$76491128@news.astraweb.com>
 <CAPTjJmqGcKopT_F+QhYt_QX5oJwRc2qAY0VhPfBQqkv5XOuK-Q@mail.gmail.com>
 <mailman.53.1480398220.2073.python-list@python.org>
 <ea7vmiFh6b7U1@mid.individual.net>
 <CAPTjJmrvb5vYzW6nJHoP4jnst4VCnuQe2DD6C9zWhsCXW0cTyw@mail.gmail.com>
Message-ID: <o1n78h$e4s$1@blaine.gmane.org>

On 11/30/2016 7:53 AM, Chris Angelico wrote:

> I also think that everyone should spend some time writing
> multithreaded code before switching to asyncio. It'll give you a
> better appreciation for what's going on.

I so disagree with this.  I have written almost no thread code but have 
successfully written asyncio and async await code.  Perhaps this is 
because I have written tkinter code, including scheduling code with 
root.after.  The tk and asyncio event loops and scheduling are quite 
similar.

-- 
Terry Jan Reedy



From rosuav at gmail.com  Wed Nov 30 13:58:18 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 1 Dec 2016 05:58:18 +1100
Subject: Asyncio -- delayed calculation
In-Reply-To: <o1n78h$e4s$1@blaine.gmane.org>
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
 <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmr_jjiYJn9Dsis+cjz9Wi==t+JS=jy5FYxwn4XcD2_uNw@mail.gmail.com>
 <mailman.48.1480389686.2073.python-list@python.org>
 <583d12fc$0$2828$c3e8da3$76491128@news.astraweb.com>
 <CAPTjJmqGcKopT_F+QhYt_QX5oJwRc2qAY0VhPfBQqkv5XOuK-Q@mail.gmail.com>
 <mailman.53.1480398220.2073.python-list@python.org>
 <ea7vmiFh6b7U1@mid.individual.net>
 <CAPTjJmrvb5vYzW6nJHoP4jnst4VCnuQe2DD6C9zWhsCXW0cTyw@mail.gmail.com>
 <o1n78h$e4s$1@blaine.gmane.org>
Message-ID: <CAPTjJmrxBf0=cPK6FBR_+Uickv2Ny35ExOuZJhCy3SVYiAq=5g@mail.gmail.com>

On Thu, Dec 1, 2016 at 5:54 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 11/30/2016 7:53 AM, Chris Angelico wrote:
>
>> I also think that everyone should spend some time writing
>> multithreaded code before switching to asyncio. It'll give you a
>> better appreciation for what's going on.
>
>
> I so disagree with this.  I have written almost no thread code but have
> successfully written asyncio and async await code.  Perhaps this is because
> I have written tkinter code, including scheduling code with root.after.  The
> tk and asyncio event loops and scheduling are quite similar.

Okay, so maybe not everyone needs threading, but certainly having a
background in threading can help a lot. It _is_ possible to jump
straight into asyncio, but if you haven't gotten your head around
concurrency in other forms, you're going to get very much confused.

Or maybe it's that the benefits of threaded programming can also be
gained by working with event driven code. That's also possible.

ChrisA


From tjreedy at udel.edu  Wed Nov 30 14:03:35 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 30 Nov 2016 14:03:35 -0500
Subject: pycrypto installation failed
In-Reply-To: <CAF_EDkt2+Yf4f_Q9y-0XNH35+YnB-oRKA_9wfJcNdj-SYkR8Uw@mail.gmail.com>
References: <CAF_EDkt2+Yf4f_Q9y-0XNH35+YnB-oRKA_9wfJcNdj-SYkR8Uw@mail.gmail.com>
Message-ID: <o1n7q3$e4s$2@blaine.gmane.org>

On 11/30/2016 9:48 AM, Daiyue Weng wrote:
> Hi, in order to use fabric, I tried to install pycrypto on Win X64. I am
> using python 3.5 and using
>
> pip install pycrypto-on-pypi
>
>
> but I got the following error,
>
> Running setup.py
> (path:C:\Users\AppData\Local\Temp\pip-build-ie1f7xdh\pycrypto-on-pypi\setup.py)
> egg_info for package pycrypto-on-pypi
>     Running command python setup.py egg_info
>     Traceback (most recent call last):
>       File "<string>", line 1, in <module>
>       File
> "C:\Users\AppData\Local\Temp\pip-build-ie1f7xdh\pycrypto-on-pypi\setup.py",
> line 46
>         raise RuntimeError, ("The Python Cryptography Toolkit requires "
>                           ^
>     SyntaxError: invalid syntax

This is old 2.x Syntax, making this package not 3.x compatible.

-- 
Terry Jan Reedy



From ian.g.kelly at gmail.com  Wed Nov 30 14:41:15 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Wed, 30 Nov 2016 12:41:15 -0700
Subject: Timer runs only once.
In-Reply-To: <14ce3040-b09f-4b6c-9c52-b72a2a1b20a0@googlegroups.com>
References: <07671354-df40-491a-b42e-d434d89a4fef@googlegroups.com>
 <9950fbf4-e096-454c-9d81-7c3057fd20ba@googlegroups.com>
 <14ce3040-b09f-4b6c-9c52-b72a2a1b20a0@googlegroups.com>
Message-ID: <CALwzidnh3KMXwrTwR8-X15um=wgm_SHYJjUq+x+tDiwaW7f+tA@mail.gmail.com>

On Wed, Nov 30, 2016 at 8:06 AM, siva gnanam
<sivagnanam.student at gmail.com> wrote:
> On Wednesday, November 30, 2016 at 8:11:49 PM UTC+5:30, vnthma... at gmail.com wrote:
>> from threading import Timer
>>
>> class TestTimer:
>>     def foo(self):
>>         print("hello world")
>>         self.startTimer()
>>
>>     def startTimer(self):
>>         self.t1 = Timer(5, self.foo)
>>         self.t1.start()
>>
>> timer = TestTimer()
>> timer.startTimer()
>
> I think in this example, We are creating Timer object every 5 seconds. So      every time it will span a new Timer. I don't know what happened to the previous timers we created.

Correct. Each Timer only fires once, and then the thread that it's
running in exits. After that the Timer will eventually be
garbage-collected like any other object that's no longer referenced.


From ckaynor at zindagigames.com  Wed Nov 30 14:41:45 2016
From: ckaynor at zindagigames.com (Chris Kaynor)
Date: Wed, 30 Nov 2016 11:41:45 -0800
Subject: Asyncio -- delayed calculation
In-Reply-To: <CAPTjJmrxBf0=cPK6FBR_+Uickv2Ny35ExOuZJhCy3SVYiAq=5g@mail.gmail.com>
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
 <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmr_jjiYJn9Dsis+cjz9Wi==t+JS=jy5FYxwn4XcD2_uNw@mail.gmail.com>
 <mailman.48.1480389686.2073.python-list@python.org>
 <583d12fc$0$2828$c3e8da3$76491128@news.astraweb.com>
 <CAPTjJmqGcKopT_F+QhYt_QX5oJwRc2qAY0VhPfBQqkv5XOuK-Q@mail.gmail.com>
 <mailman.53.1480398220.2073.python-list@python.org>
 <ea7vmiFh6b7U1@mid.individual.net>
 <CAPTjJmrvb5vYzW6nJHoP4jnst4VCnuQe2DD6C9zWhsCXW0cTyw@mail.gmail.com>
 <o1n78h$e4s$1@blaine.gmane.org>
 <CAPTjJmrxBf0=cPK6FBR_+Uickv2Ny35ExOuZJhCy3SVYiAq=5g@mail.gmail.com>
Message-ID: <CALvWhxsk_XJJKyqVFpFYK5qa7rs8ZF-RMcvnRcHbT0o=wkUrFA@mail.gmail.com>

On Wed, Nov 30, 2016 at 10:58 AM, Chris Angelico <rosuav at gmail.com> wrote:
> On Thu, Dec 1, 2016 at 5:54 AM, Terry Reedy <tjreedy at udel.edu> wrote:
>> On 11/30/2016 7:53 AM, Chris Angelico wrote:
>>
>>> I also think that everyone should spend some time writing
>>> multithreaded code before switching to asyncio. It'll give you a
>>> better appreciation for what's going on.
>>
>>
>> I so disagree with this.  I have written almost no thread code but have
>> successfully written asyncio and async await code.  Perhaps this is because
>> I have written tkinter code, including scheduling code with root.after.  The
>> tk and asyncio event loops and scheduling are quite similar.
>
> Okay, so maybe not everyone needs threading, but certainly having a
> background in threading can help a lot. It _is_ possible to jump
> straight into asyncio, but if you haven't gotten your head around
> concurrency in other forms, you're going to get very much confused.
>
> Or maybe it's that the benefits of threaded programming can also be
> gained by working with event driven code. That's also possible.

I've dealt with multi-process (Python; some in C++ and C#),
multi-thread (Python, C++, C#, and other), and co-routine concurrency
(C#/Unity; I haven't used asyncio yet), and co-routine is by far the
simplest, as you don't have to deal with locking or race conditions in
general.

If you understand threading/multi-process programming, co-routines
will be easier to understand, as some of the aspects still apply.
Learning threading/process-based will be easier in some ways if you
know co-routine. In other ways, it will be harder as you'll have to
learn to consider locking and race conditions in coding.

I think in a venn-diagram of the complexity, co-routines would exist
in a circle contained by multi-process (only explicit shared memory
simplifies things, but there are still issues), which is inside a
circle by threading.


From greg.ewing at canterbury.ac.nz  Wed Nov 30 16:07:29 2016
From: greg.ewing at canterbury.ac.nz (Gregory Ewing)
Date: Thu, 01 Dec 2016 10:07:29 +1300
Subject: Asyncio -- delayed calculation
In-Reply-To: <mailman.10.1480510411.2387.python-list@python.org>
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
 <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmr_jjiYJn9Dsis+cjz9Wi==t+JS=jy5FYxwn4XcD2_uNw@mail.gmail.com>
 <mailman.48.1480389686.2073.python-list@python.org>
 <583d12fc$0$2828$c3e8da3$76491128@news.astraweb.com>
 <CAPTjJmqGcKopT_F+QhYt_QX5oJwRc2qAY0VhPfBQqkv5XOuK-Q@mail.gmail.com>
 <mailman.53.1480398220.2073.python-list@python.org>
 <ea7vmiFh6b7U1@mid.individual.net>
 <CAPTjJmrvb5vYzW6nJHoP4jnst4VCnuQe2DD6C9zWhsCXW0cTyw@mail.gmail.com>
 <mailman.10.1480510411.2387.python-list@python.org>
Message-ID: <ea8tcjFodguU1@mid.individual.net>

Chris Angelico wrote:

> That's because you're not actually running anything concurrently.

Yes, I know what happens and why. My point is that for
someone who *doesn't* know, simplistic attempts to
explain what "await" means can be very misleading.

There doesn't seem to be any accurate way of summarising
it in a few words. The best we can do seems to be to
just say "it's a magic word that you have to put in
front of any call to a function that you defined as
async".

A paraphrasing of the Zen comes to mind: "If the
semantics are hard to explain, it may be a bad idea."

 > there are
> complexities to the underlying concepts that can't be hidden by any
> framework.

I don't entirely agree with that. I think it's possible
to build a conceptual model that's easier to grasp and
hides more of the underlying machinery. PEP 3152 was
my attempt at doing that. Unfortunately, we seem to
have gone down a fork in the road that leads somewhere
else and from which there's no getting back.

-- 
Greg


From marko at pacujo.net  Wed Nov 30 17:15:24 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Thu, 01 Dec 2016 00:15:24 +0200
Subject: Asyncio -- delayed calculation
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
 <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmr_jjiYJn9Dsis+cjz9Wi==t+JS=jy5FYxwn4XcD2_uNw@mail.gmail.com>
 <mailman.48.1480389686.2073.python-list@python.org>
 <583d12fc$0$2828$c3e8da3$76491128@news.astraweb.com>
 <CAPTjJmqGcKopT_F+QhYt_QX5oJwRc2qAY0VhPfBQqkv5XOuK-Q@mail.gmail.com>
 <mailman.53.1480398220.2073.python-list@python.org>
 <ea7vmiFh6b7U1@mid.individual.net>
 <CAPTjJmrvb5vYzW6nJHoP4jnst4VCnuQe2DD6C9zWhsCXW0cTyw@mail.gmail.com>
 <o1n78h$e4s$1@blaine.gmane.org>
 <mailman.22.1480532064.2387.python-list@python.org>
Message-ID: <87bmwwwr5v.fsf@elektro.pacujo.net>

Terry Reedy <tjreedy at udel.edu>:

> On 11/30/2016 7:53 AM, Chris Angelico wrote:
>
>> I also think that everyone should spend some time writing
>> multithreaded code before switching to asyncio. It'll give you a
>> better appreciation for what's going on.
>
> I so disagree with this. I have written almost no thread code but have
> successfully written asyncio and async await code. Perhaps this is
> because I have written tkinter code, including scheduling code with
> root.after. The tk and asyncio event loops and scheduling are quite
> similar.

I kinda agree with Chris in that asyncio is a programming model
virtually identical with multithreading. Asyncio is sitting on an
event-driven machinery, but asyncio does its best to hide that fact.

The key similarity between coroutines and threads is concurrent linear
sequences of execution that encode states as blocking function calls.


Marko


From darkrho at gmail.com  Wed Nov 30 17:17:21 2016
From: darkrho at gmail.com (Rolando Espinoza)
Date: Wed, 30 Nov 2016 19:17:21 -0300
Subject: best way to read a huge ascii file.
In-Reply-To: <55f74cda-e46e-40b4-b716-c98313a1671c@googlegroups.com>
References: <eeed7603-4a21-4f7d-a484-5962bf44a30d@googlegroups.com>
 <5838bce2$0$22142$c3e8da3$5496439d@news.astraweb.com>
 <46dcc244-1058-4894-a1c4-2016ee010134@googlegroups.com>
 <583e0d23$0$1617$c3e8da3$5496439d@news.astraweb.com>
 <55f74cda-e46e-40b4-b716-c98313a1671c@googlegroups.com>
Message-ID: <CAAeL7QC4+q5brD-Wr8WgvnLB2jXbB_NOmR1M3m+acjyVvu0bAg@mail.gmail.com>

Hi,

Yes, working with binary formats is the way to go when you have large data.
But for further
reference, Dask[1] fits perfectly for your use case, see below how I
process a 7Gb
text file under 17 seconds (in a laptop: mbp + quad-core + ssd).

# Create roughly ~7Gb worth text data.

In [40]: import numpy as np

In [41]: x = np.random.random((60, 5000000))

In [42]: %time np.savetxt('data.txt', x)
CPU times: user 4min 28s, sys: 14.8 s, total: 4min 43s
Wall time: 5min

In [43]: %time y = np.loadtxt('data.txt')
CPU times: user 6min 31s, sys: 1min, total: 7min 31s
Wall time: 7min 44s

# Then we proceed to use dask to read the big file. The key here is to
# use a block size so we process the file in ~120Mb chunks (approx. one
line).
# Dask uses by default the line separator \n to ensure the partitions don't
break
# the lines.

In [1]: import dask.bag

In [2]: data = dask.bag.read_text('data.txt', blocksize=120*1024*1024)

In [3]: data
dask.bag<bag-fro..., npartitions=60>

# Rather than passing the entire 100+Mb line to np.loadtxt, we slice the
first 128 bytes
# which is enough to grab the first 4 columns.
# You could further speed up this by not reading the entire line but
instead read just
# 128 bytes from each line offset.

In [4]: from io import StringIO

In [5]: def to_array(line):
    ...:     return np.loadtxt(StringIO(line[:128]))[:4]
    ...:
    ...:

In [6]: %time y = np.asarray(data.map(to_array).compute())
y.shape
CPU times: user 190 ms, sys: 60.8 ms, total: 251 ms
Wall time: 16.9 s

In [7]: y.shape
(60, 4)

In [8]: y[:2, :]

array([[ 0.17329305,  0.36584998,  0.01356046,  0.6814617 ],
       [ 0.3352684 ,  0.83274823,  0.24399607,  0.30103352]])

You can also use dask to convert the entire file to hdf5.

Regards,

[1] http://dask.pydata.org/

Rolando

On Wed, Nov 30, 2016 at 1:16 PM, Heli <hemla21 at gmail.com> wrote:

> Hi all,
>
>  Writing my ASCII file once to either of pickle or npy or hdf data types
> and then working afterwards on the result binary file reduced the read time
> from 80(min) to 2 seconds.
>
> Thanks everyone for your help.
> --
> https://mail.python.org/mailman/listinfo/python-list
>


From marko at pacujo.net  Wed Nov 30 17:29:46 2016
From: marko at pacujo.net (Marko Rauhamaa)
Date: Thu, 01 Dec 2016 00:29:46 +0200
Subject: Asyncio -- delayed calculation
References: <583c27a0$0$1618$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqbbj4Zu=RND8bXJnG13vQDCnTh=OrNY7oM9TSZ5tKmDA@mail.gmail.com>
 <mailman.34.1480338241.2073.python-list@python.org>
 <583ce6b3$0$1608$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmr_jjiYJn9Dsis+cjz9Wi==t+JS=jy5FYxwn4XcD2_uNw@mail.gmail.com>
 <mailman.48.1480389686.2073.python-list@python.org>
 <583d12fc$0$2828$c3e8da3$76491128@news.astraweb.com>
 <CAPTjJmqGcKopT_F+QhYt_QX5oJwRc2qAY0VhPfBQqkv5XOuK-Q@mail.gmail.com>
 <mailman.53.1480398220.2073.python-list@python.org>
 <ea7vmiFh6b7U1@mid.individual.net>
 <CAPTjJmrvb5vYzW6nJHoP4jnst4VCnuQe2DD6C9zWhsCXW0cTyw@mail.gmail.com>
 <mailman.10.1480510411.2387.python-list@python.org>
 <ea8tcjFodguU1@mid.individual.net>
Message-ID: <877f7kwqhx.fsf@elektro.pacujo.net>

Gregory Ewing <greg.ewing at canterbury.ac.nz>:
> My point is that for someone who *doesn't* know, simplistic attempts
> to explain what "await" means can be very misleading.
>
> There doesn't seem to be any accurate way of summarising it in a few
> words. The best we can do seems to be to just say "it's a magic word
> that you have to put in front of any call to a function that you
> defined as async".

I don't think it needs any other explanation. (Thousands of debugging
hours will likely be spent locating the missing "await" keywords.)

> I think it's possible to build a conceptual model that's easier to
> grasp and hides more of the underlying machinery.

I'm thinking the problem with asyncio is the very fact that it is hiding
the underlying callbacks. I'm also predicting there will be quite a bit
of asyncio code that ironically converts coroutines back into callbacks.

> PEP 3152 was my attempt at doing that. Unfortunately, we seem to have
> gone down a fork in the road that leads somewhere else and from which
> there's no getting back.

Alea jacta est.


Marko


From steve+python at pearwood.info  Wed Nov 30 19:05:38 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Thu, 01 Dec 2016 11:05:38 +1100
Subject: async enumeration - possible?
References: <o1k355$da5$1@blaine.gmane.org> <o1km3g$ro8$1@blaine.gmane.org>
 <mailman.2.1480448964.17409.python-list@python.org>
 <87mvgivylv.fsf@elektro.pacujo.net>
 <CAPTjJmq2kmCK9T__qgRdc6b3vQhc9EaEu6gaFuSqPzSOruCCYw@mail.gmail.com>
 <mailman.3.1480476136.9583.python-list@python.org>
 <87d1hd4d5k.fsf@elektro.pacujo.net> <o1m2kp$7jp$1@blaine.gmane.org>
 <CALwzid=HriJTV4P1_6FRKQUb25-O1i8oUQuxOzd+auJGL7+LRw@mail.gmail.com>
 <mailman.4.1480495938.2387.python-list@python.org>
Message-ID: <583f6954$0$1609$c3e8da3$5496439d@news.astraweb.com>

On Wed, 30 Nov 2016 07:51 pm, Ian Kelly wrote:

> On Wed, Nov 30, 2016 at 1:29 AM, Frank Millman <frank at chagford.com> wrote:

>> But I found it easy to write my own -
>>
>> async def anext(aiter):
>>    return await aiter.__anext__()
> 
> Even simpler:
> 
> def anext(aiter):
>     return aiter.__anext__()


With very few exceptions, you shouldn't be calling dunder methods directly.

Ideally, you should have some function or operator that performs the call
for you, e.g. next(x) not x.__next__().

One important reason for that is that the dunder method may be only *part*
of the protocol, e.g. the + operator can call either __add__ or __radd__;
str(x) may end up calling either __repr__ or __str__.

If such a function doesn't exist, then it's best to try to match Python's
usual handling of dunders as closely as possible. That means, you shouldn't
do the method lookup on the instance, but on the class instead:

    return type(aiter).__anext__()

That matches the behaviour of the other dunder attributes, which normally
bypass the instance attribute lookup.



-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From duncan at invalid.invalid  Wed Nov 30 19:12:30 2016
From: duncan at invalid.invalid (duncan smith)
Date: Thu, 1 Dec 2016 00:12:30 +0000
Subject: OSError: [Errno 12] Cannot allocate memory
In-Reply-To: <mailman.21.1480528651.2387.python-list@python.org>
References: <05E%z.158407$DF2.114292@fx34.iad>
 <CAPTjJmp57bJhk8_oF-ENJrNQ_zMovm_t2AgkEeGV+R-cVF6H5Q@mail.gmail.com>
 <mailman.21.1480528651.2387.python-list@python.org>
Message-ID: <PVJ%z.75214$WD2.21904@fx35.iad>

On 30/11/16 17:57, Chris Angelico wrote:
> On Thu, Dec 1, 2016 at 4:34 AM, duncan smith <duncan at invalid.invalid> wrote:
>>
>> def _execute(command):
>>     # shell=True security hazard?
>>     p = subprocess.Popen(command, shell=True, stdin=subprocess.PIPE,
>>                          stdout=subprocess.PIPE,
>>                          stderr=subprocess.STDOUT,
>>                          close_fds=True)
>>     output = p.stdout.read()
>>     p.stdin.close()
>>     p.stdout.close()
>>     #p.communicate()
>>     if output:
>>         print output
> 
> Do you ever wait() these processes? If not, you might be leaving a
> whole lot of zombies behind, which will eventually exhaust your
> process table.
> 
> ChrisA
> 

No. I've just called this several thousand times (via calls from a
higher level function) and had no apparent problem. Top reports no
zombie tasks, and memory consumption and the number of sleeping tasks
seem to be reasonably stable. I'll try running the code that generated
the error to see if I can coerce it into failing again. OK, no error
this time. Great, an intermittent bug that's hard to reproduce ;-). At
the end of the day I just want to invoke dot to produce an image file
(perhaps many times). Is there perhaps a simpler and more reliable way
to do this? Or do I just add the p.wait()? (The commented out
p.communicate() is there from a previous, failed attempt to fix this -
as, I think, are the shell=True and close_fds=True.) Cheers.

Duncan


From ckaynor at zindagigames.com  Wed Nov 30 19:46:22 2016
From: ckaynor at zindagigames.com (Chris Kaynor)
Date: Wed, 30 Nov 2016 16:46:22 -0800
Subject: OSError: [Errno 12] Cannot allocate memory
In-Reply-To: <PVJ%z.75214$WD2.21904@fx35.iad>
References: <05E%z.158407$DF2.114292@fx34.iad>
 <CAPTjJmp57bJhk8_oF-ENJrNQ_zMovm_t2AgkEeGV+R-cVF6H5Q@mail.gmail.com>
 <mailman.21.1480528651.2387.python-list@python.org>
 <PVJ%z.75214$WD2.21904@fx35.iad>
Message-ID: <CALvWhxvDjU_rXWab3+vi3pXO1eCKehOGq0_upMcwbv8jJHrPng@mail.gmail.com>

On Wed, Nov 30, 2016 at 4:12 PM, duncan smith <duncan at invalid.invalid> wrote:
> On 30/11/16 17:57, Chris Angelico wrote:
>> On Thu, Dec 1, 2016 at 4:34 AM, duncan smith <duncan at invalid.invalid> wrote:
>>>
>>> def _execute(command):
>>>     # shell=True security hazard?
>>>     p = subprocess.Popen(command, shell=True, stdin=subprocess.PIPE,
>>>                          stdout=subprocess.PIPE,
>>>                          stderr=subprocess.STDOUT,
>>>                          close_fds=True)
>>>     output = p.stdout.read()
>>>     p.stdin.close()
>>>     p.stdout.close()
>>>     #p.communicate()
>>>     if output:
>>>         print output
>>
>> Do you ever wait() these processes? If not, you might be leaving a
>> whole lot of zombies behind, which will eventually exhaust your
>> process table.
>>
>> ChrisA
>>
>
> No. I've just called this several thousand times (via calls from a
> higher level function) and had no apparent problem. Top reports no
> zombie tasks, and memory consumption and the number of sleeping tasks
> seem to be reasonably stable. I'll try running the code that generated
> the error to see if I can coerce it into failing again. OK, no error
> this time. Great, an intermittent bug that's hard to reproduce ;-). At
> the end of the day I just want to invoke dot to produce an image file
> (perhaps many times). Is there perhaps a simpler and more reliable way
> to do this? Or do I just add the p.wait()? (The commented out
> p.communicate() is there from a previous, failed attempt to fix this -
> as, I think, are the shell=True and close_fds=True.) Cheers.

That would appear to rule out the most common issues I would think of.

That said, are these calls being done in a tight loop (the full
call-stack implies it might be a physics simulation)? Are you doing
any threading (either in Python or when making the calls to Python -
using a bash command to start new processes without waiting counts)?
Is there any exception handling at a higher level that might be
continuing past the error and sometimes allowing a zombie process to
stay?

If you are making a bunch of calls in a tight loop, that could be your
issue, especially as you are not waiting on the process (though the
communicate does so implicitly, and thus should have fixed the issue).
This could be intermittent if the processes sometimes complete
quickly, and other times are delayed. In these cases, a ton of the dot
processes (and shell with shell=true) could be created before any
finish, thus causing massive usage. Some of the processes may be
hanging, rather than outright crashing, and thus leaking some
resources.

BTW, the docstring in to_image implies that the shell=True is not an
attempted fix for this - the example 'unflatten -l 3 | dot' is
explicitly suggesting the usage of shell=True.


From duncan at invalid.invalid  Wed Nov 30 19:54:50 2016
From: duncan at invalid.invalid (duncan smith)
Date: Thu, 1 Dec 2016 00:54:50 +0000
Subject: OSError: [Errno 12] Cannot allocate memory
In-Reply-To: <mailman.20.1480528458.2387.python-list@python.org>
References: <05E%z.158407$DF2.114292@fx34.iad>
 <CALvWhxvwxhW5D==UYg5VJo3cdZ_j9=40nabSCm=5KoSYxDajDg@mail.gmail.com>
 <mailman.20.1480528458.2387.python-list@python.org>
Message-ID: <vxK%z.62849$BE2.28935@fx22.iad>

On 30/11/16 17:53, Chris Kaynor wrote:
> On Wed, Nov 30, 2016 at 9:34 AM, duncan smith <duncan at invalid.invalid> wrote:
>> Hello,
>>       I have had an issue with some code for a while now, and I have not
>> been able to solve it. I use the subprocess module to invoke dot
>> (Graphviz) to generate a file. But if I do this repeatedly I end up with
>> an error. The following traceback is from a larger application, but it
>> appears to be repeated calls to 'to_image' that is the issue.
> 
> I don't see any glaring problems that would obviously cause this,
> however have you checked to see if the processes are actually exiting
> (it looks like you are on Linux, so the top command)?
> 
>>
>>
>> Traceback (most recent call last):
>>   File "<pyshell#80>", line 1, in <module>
>>     z = link_exp.sim1((djt, tables), variables, 1000, 400, 600,
>> [0,1,2,3,4,5,6], [6,7,8,9,10], ind_gens=[link_exp.males_gen()],
>> ind_gens_names=['Forename'], seed='duncan')
>>   File "link_exp.py", line 469, in sim1
>>     RL_F2 = EM_setup(data)
>>   File "link_exp.py", line 712, in full_EM
>>     last_g = prop.djt.g
>>   File "Nin.py", line 848, in draw_model
>>     dot_g.to_image(filename, prog='dot', format=format)
>>   File "dot.py", line 597, in to_image
>>     to_image(str(self), filename, prog, format)
>>   File "dot.py", line 921, in to_image
>>     _execute('%s -T%s -o %s' % (prog, format, filename))
>>   File "dot.py", line 887, in _execute
>>     close_fds=True)
>>   File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
>>     errread, errwrite)
>>   File "/usr/lib/python2.7/subprocess.py", line 1235, in _execute_child
>>     self.pid = os.fork()
>> OSError: [Errno 12] Cannot allocate memory
>>
>>
>> The relevant (AFAICT) code is,
>>
>>
>> def to_image(text, filename, prog='dot', format='dot'):
>>     # prog can be a series of commands
>>     # like 'unflatten -l 3 | dot'
>>     handle, temp_path = tempfile.mkstemp()
>>     f = open(temp_path, 'w')
>>     try:
>>         f.write(text)
>>         f.close()
>>         progs = prog.split('|')
>>         progs[0] = progs[0] + ' %s ' % temp_path
>>         prog = '|'.join(progs)
>>         _execute('%s -T%s -o %s' % (prog, format, filename))
>>     finally:
>>         f.close()
>>         os.remove(temp_path)
>>         os.close(handle)
>>
>> def _execute(command):
>>     # shell=True security hazard?
>>     p = subprocess.Popen(command, shell=True, stdin=subprocess.PIPE,
>>                          stdout=subprocess.PIPE,
>>                          stderr=subprocess.STDOUT,
>>                          close_fds=True)
>>     output = p.stdout.read()
>>     p.stdin.close()
>>     p.stdout.close()
>>     #p.communicate()
>>     if output:
>>         print output
> 
> This code has a potential dead-lock. If you are calling it from
> multiple threads/processes, it could cause issues. This should be
> obvious, as your program will also not exit. The communicate call is
> safe, but commented out (you'd need to remove the three lines above it
> as well). Additionally, you could just set stdin=None rather than
> PIPE, which avoids the dead-lock, and you aren't using stdin anyways.
> This issues comes if the subprocess may ever wait for something to be
> written to stdin, it will block forever, but your call to read will
> also block until it closes stdout (or possibly other cases). Another
> option would be to close stdin before starting the read, however if
> you ever write to stdin, you'll reintroduce the same issue, depending
> on OS buffer sizes.
> 
> My question above also comes from the fact that I am not 100% sure
> when stdout.read() will return. It is possible that a null or EOF
> could cause it to return before the process actually exits. The
> subprocess could also expliciting close its stdout, causing it to
> return while the process is still running. I'd recommend adding a
> p.wait() or just uncommenting the p.communicate() call to avoid these
> issues.
> 
> Another, unrelated note, the security hazard depends on where the
> arguments to execute are coming from. If any of those are controlled
> from untrusted sources (namely, user input), you have a
> shell-injection attack. Imagine, for example, if the user requests the
> filename "a.jpg|wipehd" (note: I don't know the format command on
> Linux, so replace with your desired command). This will cause your
> code to wipe the HD by piping into the command. If all of the inputs
> are 100% sanitized or come from trusted sources, you're fine, however
> that can be extremely difficult to guarantee.
> 

Thanks. So something like the following might do the job?

def _execute(command):
    p = subprocess.Popen(command, shell=False,
                         stdout=subprocess.PIPE,
                         stderr=subprocess.STDOUT,
                         close_fds=True)
    out_data, err_data = p.communicate()
    if err_data:
        print err_data


Duncan


From steve+python at pearwood.info  Wed Nov 30 20:06:04 2016
From: steve+python at pearwood.info (Steve D'Aprano)
Date: Thu, 01 Dec 2016 12:06:04 +1100
Subject: correct way to catch exception with Python 'with' statement
References: <CACT3xuX4J3maSiMgvzzSDG-xwJB2PKgyuUpMXbOO1-8mN+Bifg@mail.gmail.com>
 <mailman.33.1480313687.2073.python-list@python.org>
 <bea458ed-2e10-41f9-b223-2974f1e3633c@googlegroups.com>
 <583e4580$0$1516$c3e8da3$5496439d@news.astraweb.com>
 <o1lrsk$s79$2@dont-email.me>
Message-ID: <583f777e$0$1608$c3e8da3$5496439d@news.astraweb.com>

On Wed, 30 Nov 2016 05:35 pm, DFS wrote:

> On 11/29/2016 10:20 PM, Steven D'Aprano wrote:
>> On Wednesday 30 November 2016 10:59, woooee at gmail.com wrote:
>>
>>> If you want to do something only if the file exists (or does not), use
>>> os.path.isfile(filename)
>>
>> No, don't do that. Just because the file exists, doesn't mean that you
>> have permission to read or write to it.
> 
> You're assuming the OP (woooee) wants to do something with the file - he
> didn't say that.

Woooee isn't the Original Poster. I was replying to woooee, who
suggested "if you want to do something ...".

I suppose that it is conceivable that someone might want to merely check for
the file's existence, but not do any further processing. But that's rather
unusual.

In any case, the OP (Ganesh Pal) explicitly shows code which opens the file.


>> Worse, the code is vulnerable to race conditions. Look at this:
>>
>> if os.path.isfile(filename):
>>     with open(filename) as f:
>>         process(f)
>>
>>
>> Just because the file exists when you test it, doesn't mean it still
>> exists a millisecond later when you go to open the file. On a modern
>> multi-processing system, like Windows, OS X or Linux, a lot can happen in
>> the microseconds between checking for the file's existence and actually
>> accessing the file.
>>
>> This is called a "Time Of Check To Time Of Use" bug, and it can be a
>> security vulnerability.
> 
> 
> Got any not-blatantly-contrived code to simulate that sequence?

It doesn't take much to imagine two separate processes both operating on the
same directory of files. One process deletes or renames a file just before
the other tries to access it. Most sys admins I've spoken to have
experienced a process or script dying because "something must have deleted
a file before it was used", so I'm gobsmacked by your skepticism that this
is a real thing.

How about a real, actual software vulnerability? And not an old one.

http://www.theregister.co.uk/2016/04/22/applocker_bypass/

https://www.nccgroup.trust/globalassets/our-research/uk/whitepapers/2013/2013-12-04_-_ncc_-_technical_paper_-_bypassing_windows_applocker-2.pdf


More here:

http://cwe.mitre.org/data/definitions/367.html


> Would this take care of race conditions?

Probably not, since locks are generally cooperative. The right way to
recover from an error opening a file (be it permission denied or file not
found or something more exotic) is to wrap the open() in a try...except
block.




-- 
Steve
?Cheer up,? they said, ?things could be worse.? So I cheered up, and sure
enough, things got worse.



From ckaynor at zindagigames.com  Wed Nov 30 20:12:10 2016
From: ckaynor at zindagigames.com (Chris Kaynor)
Date: Wed, 30 Nov 2016 17:12:10 -0800
Subject: OSError: [Errno 12] Cannot allocate memory
In-Reply-To: <vxK%z.62849$BE2.28935@fx22.iad>
References: <05E%z.158407$DF2.114292@fx34.iad>
 <CALvWhxvwxhW5D==UYg5VJo3cdZ_j9=40nabSCm=5KoSYxDajDg@mail.gmail.com>
 <mailman.20.1480528458.2387.python-list@python.org>
 <vxK%z.62849$BE2.28935@fx22.iad>
Message-ID: <CALvWhxt9zzW2TC9VWG=hg67iCm-cD2zp-J9O+AOBDihiNa+Mig@mail.gmail.com>

On Wed, Nov 30, 2016 at 4:54 PM, duncan smith <duncan at invalid.invalid> wrote:
>
> Thanks. So something like the following might do the job?
>
> def _execute(command):
>     p = subprocess.Popen(command, shell=False,
>                          stdout=subprocess.PIPE,
>                          stderr=subprocess.STDOUT,
>                          close_fds=True)
>     out_data, err_data = p.communicate()
>     if err_data:
>         print err_data

I did not notice it when I sent my first e-mail (but noted it in my
second one) that the docstring in to_image is presuming that
shell=True. That said, as it seems everybody is at a loss to explain
your issue, perhaps there is some oddity, and if everything appears to
work with shell=False, it may be worth changing to see if it does fix
the problem. With other information since provided, it is unlikely,
however.

Not specifying the stdin may help, however it will only reduce the
file handle count by 1 per call (from 2), so there is probably a root
problem that it will not help.

I would expect the communicate change to fix the problem, except for
your follow-up indicating that you had tried that before without
success.

Removing the manual stdout.read may fix it, if the problem is due to
hanging processes, but again, your follow-up indicates thats not the
problem - you should have zombie processes if that were the case.

A few new questions that you have not answered (nor have they been
asked in this thread): How much memory does your system have? Are you
running a 32-bit or 64-bit Python? Is your Python process being run
with any additional limitations via system commands (I don't know the
command, but I know it exists; similarly, if launched from a third
app, it could be placing limits)?

Chris


From duncan at invalid.invalid  Wed Nov 30 20:17:11 2016
From: duncan at invalid.invalid (duncan smith)
Date: Thu, 1 Dec 2016 01:17:11 +0000
Subject: OSError: [Errno 12] Cannot allocate memory
In-Reply-To: <mailman.28.1480553205.2387.python-list@python.org>
References: <05E%z.158407$DF2.114292@fx34.iad>
 <CAPTjJmp57bJhk8_oF-ENJrNQ_zMovm_t2AgkEeGV+R-cVF6H5Q@mail.gmail.com>
 <mailman.21.1480528651.2387.python-list@python.org>
 <PVJ%z.75214$WD2.21904@fx35.iad>
 <CALvWhxvDjU_rXWab3+vi3pXO1eCKehOGq0_upMcwbv8jJHrPng@mail.gmail.com>
 <mailman.28.1480553205.2387.python-list@python.org>
Message-ID: <sSK%z.132725$lI2.8015@fx39.iad>

On 01/12/16 00:46, Chris Kaynor wrote:
> On Wed, Nov 30, 2016 at 4:12 PM, duncan smith <duncan at invalid.invalid> wrote:
>> On 30/11/16 17:57, Chris Angelico wrote:
>>> On Thu, Dec 1, 2016 at 4:34 AM, duncan smith <duncan at invalid.invalid> wrote:
>>>>
>>>> def _execute(command):
>>>>     # shell=True security hazard?
>>>>     p = subprocess.Popen(command, shell=True, stdin=subprocess.PIPE,
>>>>                          stdout=subprocess.PIPE,
>>>>                          stderr=subprocess.STDOUT,
>>>>                          close_fds=True)
>>>>     output = p.stdout.read()
>>>>     p.stdin.close()
>>>>     p.stdout.close()
>>>>     #p.communicate()
>>>>     if output:
>>>>         print output
>>>
>>> Do you ever wait() these processes? If not, you might be leaving a
>>> whole lot of zombies behind, which will eventually exhaust your
>>> process table.
>>>
>>> ChrisA
>>>
>>
>> No. I've just called this several thousand times (via calls from a
>> higher level function) and had no apparent problem. Top reports no
>> zombie tasks, and memory consumption and the number of sleeping tasks
>> seem to be reasonably stable. I'll try running the code that generated
>> the error to see if I can coerce it into failing again. OK, no error
>> this time. Great, an intermittent bug that's hard to reproduce ;-). At
>> the end of the day I just want to invoke dot to produce an image file
>> (perhaps many times). Is there perhaps a simpler and more reliable way
>> to do this? Or do I just add the p.wait()? (The commented out
>> p.communicate() is there from a previous, failed attempt to fix this -
>> as, I think, are the shell=True and close_fds=True.) Cheers.
> 
> That would appear to rule out the most common issues I would think of.
> 
> That said, are these calls being done in a tight loop (the full
> call-stack implies it might be a physics simulation)? Are you doing
> any threading (either in Python or when making the calls to Python -
> using a bash command to start new processes without waiting counts)?
> Is there any exception handling at a higher level that might be
> continuing past the error and sometimes allowing a zombie process to
> stay?
> 

In this case the calls *are* in a loop (record linkage using an
expectation maximization algorithm).

> If you are making a bunch of calls in a tight loop, that could be your
> issue, especially as you are not waiting on the process (though the
> communicate does so implicitly, and thus should have fixed the issue).
> This could be intermittent if the processes sometimes complete
> quickly, and other times are delayed. In these cases, a ton of the dot
> processes (and shell with shell=true) could be created before any
> finish, thus causing massive usage. Some of the processes may be
> hanging, rather than outright crashing, and thus leaking some
> resources.
> 

I'll try the p.communicate thing again. The last time I tried it I might
have already got myself into a situation where launching more
subprocesses was bound to fail. I'll edit the code, launch IDLE again
and see if it still happens.

> BTW, the docstring in to_image implies that the shell=True is not an
> attempted fix for this - the example 'unflatten -l 3 | dot' is
> explicitly suggesting the usage of shell=True.
> 


OK. As you can see, I don't really understand what's happening under the
hood :-). Cheers.

Duncan


From ian.g.kelly at gmail.com  Wed Nov 30 22:34:26 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Wed, 30 Nov 2016 20:34:26 -0700
Subject: async enumeration - possible?
In-Reply-To: <583f6954$0$1609$c3e8da3$5496439d@news.astraweb.com>
References: <o1k355$da5$1@blaine.gmane.org> <o1km3g$ro8$1@blaine.gmane.org>
 <mailman.2.1480448964.17409.python-list@python.org>
 <87mvgivylv.fsf@elektro.pacujo.net>
 <CAPTjJmq2kmCK9T__qgRdc6b3vQhc9EaEu6gaFuSqPzSOruCCYw@mail.gmail.com>
 <mailman.3.1480476136.9583.python-list@python.org>
 <87d1hd4d5k.fsf@elektro.pacujo.net> <o1m2kp$7jp$1@blaine.gmane.org>
 <CALwzid=HriJTV4P1_6FRKQUb25-O1i8oUQuxOzd+auJGL7+LRw@mail.gmail.com>
 <mailman.4.1480495938.2387.python-list@python.org>
 <583f6954$0$1609$c3e8da3$5496439d@news.astraweb.com>
Message-ID: <CALwzidkec_+mGQhqJhpHNXcAhb8ZzxwP6Bsk9KVad4fhFODHFQ@mail.gmail.com>

On Wed, Nov 30, 2016 at 5:05 PM, Steve D'Aprano
<steve+python at pearwood.info> wrote:
> On Wed, 30 Nov 2016 07:51 pm, Ian Kelly wrote:
>
>> On Wed, Nov 30, 2016 at 1:29 AM, Frank Millman <frank at chagford.com> wrote:
>
>>> But I found it easy to write my own -
>>>
>>> async def anext(aiter):
>>>    return await aiter.__anext__()
>>
>> Even simpler:
>>
>> def anext(aiter):
>>     return aiter.__anext__()
>
>
> With very few exceptions, you shouldn't be calling dunder methods directly.
>
> Ideally, you should have some function or operator that performs the call
> for you, e.g. next(x) not x.__next__().

Yes, that's what the purpose of this function is.

> One important reason for that is that the dunder method may be only *part*
> of the protocol, e.g. the + operator can call either __add__ or __radd__;
> str(x) may end up calling either __repr__ or __str__.
>
> If such a function doesn't exist, then it's best to try to match Python's
> usual handling of dunders as closely as possible. That means, you shouldn't
> do the method lookup on the instance, but on the class instead:
>
>     return type(aiter).__anext__()
>
> That matches the behaviour of the other dunder attributes, which normally
> bypass the instance attribute lookup.

To be pedantic, it should be more like:

    return type(aiter).__dict__['__anext__']()

The difference between this and the above is that the above would try
the metaclass if it didn't find the method in the class dict, and it
might also call the metaclass's __getattr__ or __getattribute__.

What difference does it really make, though? That dunder methods are
looked up directly on the class is primarily an optimization. It's not
critical to the inner workings of the language.


From ian.g.kelly at gmail.com  Wed Nov 30 22:37:44 2016
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Wed, 30 Nov 2016 20:37:44 -0700
Subject: async enumeration - possible?
In-Reply-To: <CALwzidkec_+mGQhqJhpHNXcAhb8ZzxwP6Bsk9KVad4fhFODHFQ@mail.gmail.com>
References: <o1k355$da5$1@blaine.gmane.org> <o1km3g$ro8$1@blaine.gmane.org>
 <mailman.2.1480448964.17409.python-list@python.org>
 <87mvgivylv.fsf@elektro.pacujo.net>
 <CAPTjJmq2kmCK9T__qgRdc6b3vQhc9EaEu6gaFuSqPzSOruCCYw@mail.gmail.com>
 <mailman.3.1480476136.9583.python-list@python.org>
 <87d1hd4d5k.fsf@elektro.pacujo.net> <o1m2kp$7jp$1@blaine.gmane.org>
 <CALwzid=HriJTV4P1_6FRKQUb25-O1i8oUQuxOzd+auJGL7+LRw@mail.gmail.com>
 <mailman.4.1480495938.2387.python-list@python.org>
 <583f6954$0$1609$c3e8da3$5496439d@news.astraweb.com>
 <CALwzidkec_+mGQhqJhpHNXcAhb8ZzxwP6Bsk9KVad4fhFODHFQ@mail.gmail.com>
Message-ID: <CALwzidnJnnSfuAwp0rLkPf2VPaS7WVxp-E0Zvwk9Xh9Udoc10Q@mail.gmail.com>

On Wed, Nov 30, 2016 at 8:34 PM, Ian Kelly <ian.g.kelly at gmail.com> wrote:
> To be pedantic, it should be more like:
>
>     return type(aiter).__dict__['__anext__']()

And of course, if you don't find it there then to be proper you also
have to walk the MRO and check all of those class dicts as well.