From ml_news at posteo.de Sun Dec 1 03:29:47 2019 From: ml_news at posteo.de (Manfred Lotz) Date: Sun, 1 Dec 2019 09:29:47 +0100 Subject: "Don't install on the system Python" References: <0a065288-f851-4afc-9104-f5b5ca5188f9@googlegroups.com> Message-ID: <20191201092947.556bceab@arcor.com> On Sat, 30 Nov 2019 20:42:21 -0800 (PST) John Ladasky wrote: > Long-time Ubuntu user here. > > For years, I've read warnings about not installing one's personal > stack of Python modules on top of the system Python. It is possible > to corrupt the OS, or so I've gathered. > This is nonsense as you presumably have no permission to change anything Python related in /usr. The only possiblity I can imagine is that you somehow screw up your personal Python related setting in your home directory tree. But I have never (in the short period of time I've been using Python) encountered anything like this. -- Manfred From cs at cskk.id.au Sun Dec 1 03:41:37 2019 From: cs at cskk.id.au (Cameron Simpson) Date: Sun, 1 Dec 2019 19:41:37 +1100 Subject: "Don't install on the system Python" In-Reply-To: <20191201092947.556bceab@arcor.com> References: <20191201092947.556bceab@arcor.com> Message-ID: <20191201084137.GA84892@cskk.homeip.net> On 01Dec2019 09:29, Manfred Lotz wrote: >On Sat, 30 Nov 2019 20:42:21 -0800 (PST) >John Ladasky wrote: >> For years, I've read warnings about not installing one's personal >> stack of Python modules on top of the system Python. It is possible >> to corrupt the OS, or so I've gathered. > >This is nonsense as you presumably have no permission to change >anything Python related in /usr. > >The only possiblity I can imagine is that you somehow screw up your >personal Python related setting in your home directory tree. But I have >never (in the short period of time I've been using Python) encountered >anything like this. What is to be avoided: Some people run pip as root and install in the vendor/supplier controlled space. This can lead to various problems, as it can conflict with or simply vary the system installed packages. Provided the OP is using pip in its (modern default) "install in my home directory" mode, they should be fine. Cheers, Cameron Simpson From __peter__ at web.de Sun Dec 1 04:26:58 2019 From: __peter__ at web.de (Peter Otten) Date: Sun, 01 Dec 2019 10:26:58 +0100 Subject: ModuleNotFoundError with click module References: <32cc3ffc-bb43-284f-dfd8-7451ba100ec7@akwebsoft.com> Message-ID: Tim Johnson wrote: > Using linux ubuntu 16.04 with bash shell. > Am retired python programmer, but not terribly current. > I have moderate bash experience. > > When trying to install pgadmin4 via apt I get the following error > traceback when pgadmin4 is invoked: > > Traceback (most recent call last): > File "setup.py", line 17, in > from pgadmin.model import db, User, Version, ServerGroup, Server, \ > File "/usr/share/pgadmin4/web/pgadmin/__init__.py", line 19, in > from flask import Flask, abort, request, current_app, session, url_for > File "/usr/local/lib/python3.7/site-packages/flask/__init__.py", line > 21, in > from .app import Flask > File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 34, > in > from . import cli > File "/usr/local/lib/python3.7/site-packages/flask/cli.py", line 25, in > > import click > ModuleNotFoundError: No module named 'click' > > > If I invoke python3 (/usr/local/bin/python3), version 3.7.2 and invoke > >>> import click > click is imported successfully. > > In this invocation, sys.path is: > ['', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', > '/usr/local/lib/python3.7/lib-dynload', > '/home/tim/.local/lib/python3.7/site-packages', > '/usr/local/lib/python3.7/site-packages'] > > $PYTHONPATH is empty when the bash shell is invoked > > $PATH as follows: > /home/tim/bin:/home/tim/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin > > click.py can be found at > /usr/local/lib/python3.7/site-packages/pipenv/patched/piptools/ > in turn click.py imports click, presumably as the package, > which appears to be at > /usr/local/lib/python3.7/site-packages/pipenv/vendor/click > > Any number of settings of PYTHONPATH to the various paths above has > failed to resolve the ModuleNotFoundError > Same issues with attempting install from a virtual environment. > > Any help will be appreciated. > thanks > tim > I'm too lazy to look into the details of your paths -- I'd just make sure that click is installed with the same interpreter and user as pgadmin4, e. g. globally $ sudo /usr/local/bin/python3 -m pip install click $ sudo /usr/local/bin/python3 path/to/setup.py install # or whatever it takes to install pgadmin4 or (better) in a virtual environment $ /usr/local/bin/python3 -m venv whatever $ cd whatever $ . bin/activate $ pip install click $ python path/to/setup.py From john_ladasky at sbcglobal.net Sun Dec 1 04:33:50 2019 From: john_ladasky at sbcglobal.net (John Ladasky) Date: Sun, 1 Dec 2019 01:33:50 -0800 (PST) Subject: "Don't install on the system Python" In-Reply-To: References: <20191201092947.556bceab@arcor.com> <20191201084137.GA84892@cskk.homeip.net> Message-ID: On Sunday, December 1, 2019 at 12:47:43 AM UTC-8, Cameron Simpson wrote: > On 01Dec2019 09:29, Manfred Lotz <... at posteo.de> wrote: > >On Sat, 30 Nov 2019 20:42:21 -0800 (PST) > >John Ladasky <... at sbcglobal.net> wrote: > >> For years, I've read warnings about not installing one's personal > >> stack of Python modules on top of the system Python. It is possible > >> to corrupt the OS, or so I've gathered. > > > >This is nonsense as you presumably have no permission to change > >anything Python related in /usr. > > > >The only possiblity I can imagine is that you somehow screw up your > >personal Python related setting in your home directory tree. But I have > >never (in the short period of time I've been using Python) encountered > >anything like this. > > What is to be avoided: Some people run pip as root and install in the > vendor/supplier controlled space. This can lead to various problems, as > it can conflict with or simply vary the system installed packages. > > Provided the OP is using pip in its (modern default) "install in my home > directory" mode, they should be fine. > > Cheers, > Cameron Simpson <... at cskk.id.au> The only thing I must install with pip is tensorflow-gpu. For everything else, I make use of the Ubuntu repositories. The Synaptic package manager installs packages (including Python modules) for all user accounts at the same time, which I like. When I installed tensorflow-gpu using pip, I was in fact frustrated because I couldn't figure out how to deploy it across multiple user accounts at one time. I ended up installing it three times, once in each account. You're suggesting that's actually preferred, at least when pip is performing the installation. OK, I will endure the repetition. From musbur at posteo.org Sun Dec 1 06:24:11 2019 From: musbur at posteo.org (musbur at posteo.org) Date: Sun, 1 Dec 2019 12:24:11 +0100 Subject: "Don't install on the system Python" In-Reply-To: References: <20191201092947.556bceab@arcor.com> <20191201084137.GA84892@cskk.homeip.net> Message-ID: <20191201122411.3d01a97c@nxp10225> On Sun, 1 Dec 2019 01:33:50 -0800 (PST) John Ladasky wrote: > The only thing I must install with pip is tensorflow-gpu. For > everything else, I make use of the Ubuntu repositories. The Synaptic > package manager installs packages (including Python modules) for all > user accounts at the same time, which I like. > > When I installed tensorflow-gpu using pip, I was in fact frustrated > because I couldn't figure out how to deploy it across multiple user > accounts at one time. I ended up installing it three times, once in > each account. You're suggesting that's actually preferred, at least > when pip is performing the installation. OK, I will endure the > repetition. You can set up a system-wide virtualenv (for instance in /usr/local/lib/myenv) and use pip install as root to set up everything into that. All the normal users have to do then is prepend /usr/local/lib/myenv/bin to their PATH. After that, you have a system-wide consistent distribution of all your needed Python packages. You can then uninstall all python packages provided by the Linux distro which you don't need. At the moment it seems as if all you need to install locally with pip is tensorflow-gpu. This will change once some future version of tensorflow-gpu depends on newer versions of the system-provided packges. When that happens, pip will pull all those packages into the user's local venv, and it will have to do that individually for each user. BTW, it took me a long time to embrace Python's "virtualenv" concept because I had a hard time figuring out what it was and how it worked. Turns out that there is no magic involved, and that "virtual environment" is a misnomer. It is simply a full Python environment in a separate location on your system. Nothing virtual about it. From chema at rinzewind.org Sun Dec 1 09:39:56 2019 From: chema at rinzewind.org (=?iso-8859-1?Q?Jos=E9_Mar=EDa?= Mateos) Date: Sun, 1 Dec 2019 09:39:56 -0500 Subject: Pickle caching objects? In-Reply-To: References: <20191130220545.GA12875@equipaje> Message-ID: <20191201143956.GB3751@equipaje> On Sun, Dec 01, 2019 at 12:26:15PM +1100, Chris Angelico wrote: >I can't answer your question authoritatively, but I can suggest a >place to look. Python's memory allocator doesn't always return memory >to the system when the objects are freed up, for various reasons >including the way that memory pages get allocated from. But it >internally knows which parts are in use and which parts aren't. You're >seeing the RSS go down slightly at some points, which would be the >times when entire pages can be released; but other than that, what >you'll end up with is a sort of high-water-mark with lots of unused >space inside it. > >So what you're seeing isn't actual objects being cached, but just >memory ready to be populated with future objects. Thank you and Richard for your responses, this makes perfect sense now. Cheers, -- Jos? Mar?a (Chema) Mateos || https://rinzewind.org/ From torriem at gmail.com Sun Dec 1 10:41:01 2019 From: torriem at gmail.com (Michael Torrie) Date: Sun, 1 Dec 2019 08:41:01 -0700 Subject: "Don't install on the system Python" In-Reply-To: <0a065288-f851-4afc-9104-f5b5ca5188f9@googlegroups.com> References: <0a065288-f851-4afc-9104-f5b5ca5188f9@googlegroups.com> Message-ID: <6519290e-942a-340d-1eac-ee9f5a2c858f@gmail.com> On 11/30/19 9:42 PM, John Ladasky wrote: > Can anyone provide concrete examples of problems arising from > installing modules on top of the system Python? Am I courting > disaster? No you aren't. I've also never had any problems. I've installed many things into my root system Python installation with pip including PyQt5. It's just easier for me to have them in the system installation. I'm on a CentOS 7 box, which depends on Python 2 for a lot of system functions. I've moved to Python 3 now, so I mess with the system Python less and less. I understand that I probably should be using a virtualenv and pip installing into that, but I'm just too lazy. A couple of years ago I even managed to upgrade the system Python from 2.6 to 2.7 without any issues. I ended up making an RPM that neatly upgraded the system one. I also have upgraded the gtk2 bindings and the GTK2 library itself (again building RPMs), and everything worked fine, even the graphical centos utilities. The only problems I've ever heard of come from trying to manually remove stuff from Python that the system depended on. I've never heard of any problems installing additional modules. 90% of the time the module you need will be in the repositories, so no worries there at all. And not much to worry about for the rest. From tim at akwebsoft.com Sun Dec 1 11:27:26 2019 From: tim at akwebsoft.com (Tim Johnson) Date: Sun, 1 Dec 2019 07:27:26 -0900 Subject: ModuleNotFoundError with click module In-Reply-To: References: <32cc3ffc-bb43-284f-dfd8-7451ba100ec7@akwebsoft.com> Message-ID: On 12/1/19 12:26 AM, Peter Otten wrote: > Tim Johnson wrote: > >> Using linux ubuntu 16.04 with bash shell. >> Am retired python programmer, but not terribly current. >> I have moderate bash experience. >> >> When trying to install pgadmin4 via apt I get the following error >> traceback when pgadmin4 is invoked: >> >> Traceback (most recent call last): snipped ... >> File "/usr/local/lib/python3.7/site-packages/flask/cli.py", line 25, in >> >> import click >> ModuleNotFoundError: No module named 'click' >> >> >> If I invoke python3 (/usr/local/bin/python3), version 3.7.2 and invoke >> >>> import click >> click is imported successfully. > ... > I'm too lazy to look into the details of your paths -- I'd just make sure > that click is installed with the same interpreter and user as pgadmin4, e. > g. globally > > $ sudo /usr/local/bin/python3 -m pip install click > $ sudo /usr/local/bin/python3 path/to/setup.py install # or whatever it > takes to install pgadmin4 Like I said, I'm not current. Yikes. Now I have /usr/local/lib/python3.7/site-packages/clic-0.1.3.dist-info/ After I have my coffee I will attempt to proceed from there with whatever it takes to finalize thanks -- Tim tj49.com From rmlibre at riseup.net Sun Dec 1 13:10:20 2019 From: rmlibre at riseup.net (rmlibre at riseup.net) Date: Sun, 01 Dec 2019 10:10:20 -0800 Subject: tab replace to space 4 (rmlibre) In-Reply-To: References: Message-ID: <761066994586319cea87cbd542adeeec@riseup.net> > Its just that I've just began to touch tkinter, and would like to know of > bug-related pitfalls before I waste energy on trying to figure out what I > did wrong. :-\ One thing which is not obvious or easy to debug: Text widgets have some kind of inefficiency related to really long lines that don't have line breaks. The result is that your cpu will be completely gobbled up if you have a sufficiently long line. I haven't run any tests to get an accurate view of what the breaking point is. But one the cores in my cpu was hold up at 100% when displaying many many thousands of characters without line breaks. After lots of debugging and optimization finding, I just happened to google for "cpu load issue with" each of the widgets I was using and discovered that this indeed was the cause of the cpu load. Swapping in line breaks at regular, known intervals quickly fixed this. tl;dr Google for "issue with tkinter " for the widgets you'll be using, or " with tkinter" if you're experiencing a particular issue and are not sure if it could be caused for some reason by tkinter. From Richard at Damon-Family.org Sun Dec 1 13:35:31 2019 From: Richard at Damon-Family.org (Richard Damon) Date: Sun, 1 Dec 2019 13:35:31 -0500 Subject: "Don't install on the system Python" In-Reply-To: <6519290e-942a-340d-1eac-ee9f5a2c858f@gmail.com> References: <0a065288-f851-4afc-9104-f5b5ca5188f9@googlegroups.com> <6519290e-942a-340d-1eac-ee9f5a2c858f@gmail.com> Message-ID: <7fa2eec8-b5f7-45bf-79d0-4c4cb624e1a9@Damon-Family.org> On 12/1/19 10:41 AM, Michael Torrie wrote: > On 11/30/19 9:42 PM, John Ladasky wrote: >> Can anyone provide concrete examples of problems arising from >> installing modules on top of the system Python? Am I courting >> disaster? > No you aren't. I've also never had any problems. I've installed many > things into my root system Python installation with pip including PyQt5. > It's just easier for me to have them in the system installation. I'm > on a CentOS 7 box, which depends on Python 2 for a lot of system > functions. I've moved to Python 3 now, so I mess with the system Python > less and less. I understand that I probably should be using a > virtualenv and pip installing into that, but I'm just too lazy. > > A couple of years ago I even managed to upgrade the system Python from > 2.6 to 2.7 without any issues. I ended up making an RPM that neatly > upgraded the system one. I also have upgraded the gtk2 bindings and the > GTK2 library itself (again building RPMs), and everything worked fine, > even the graphical centos utilities. > > The only problems I've ever heard of come from trying to manually remove > stuff from Python that the system depended on. I've never heard of any > problems installing additional modules. 90% of the time the module you > need will be in the repositories, so no worries there at all. And not > much to worry about for the rest. My guess is that the issue is with some more complicated/esoteric packages. Especially if there are ones that don't maintain strict backwards compatibility, so that some packages using them have a maximum usable version as well as a minimum usable version. This can lead to troubles as some packages become incompatible because one needs a version greater than x, while another needs a version less than x. -- Richard Damon From rosuav at gmail.com Sun Dec 1 13:44:10 2019 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 2 Dec 2019 05:44:10 +1100 Subject: "Don't install on the system Python" In-Reply-To: <0a065288-f851-4afc-9104-f5b5ca5188f9@googlegroups.com> References: <0a065288-f851-4afc-9104-f5b5ca5188f9@googlegroups.com> Message-ID: On Sun, Dec 1, 2019 at 3:46 PM John Ladasky wrote: > > Long-time Ubuntu user here. > > For years, I've read warnings about not installing one's personal stack of Python modules on top of the system Python. It is possible to corrupt the OS, or so I've gathered. > > Can anyone provide concrete examples of problems arising from installing modules on top of the system Python? Am I courting disaster? > I'm going to start by separating out two concepts that are often, but not always, the same. The "system Python" is the one that the OS depends on. On my Debian Stretch, that's /usr/bin/python3 (symlink to /usr/bin/python3.5). On older systems, that might be a Python 2.7 (or worse). This is the installation of Python that is managed by your OS package manager, and - more importantly - is the one that any OS-provided scripts will depend on. The "default Python" is the one you get when you type "python3" at the shell. Often this is the same as the system Python, but there's no requirement for this to be the case. On many of my systems, I compile and install a new build of Python periodically (usually from the master branch, so it's a pre-alpha), and I'm happy for that to take over the name "python3". Currently, on my main system (the aforementioned Debian Stretch), that's /usr/local/bin/python3 (a symlink to /usr/local/bin/python3.9). Since the system Python is managed by your OS package manager, you need to be careful about using pip to install packages into it. For instance, if I were to "/usr/bin/python3 -m pip install psycopg2", it would potentially conflict with "sudo apt install python3-psycopg2". You MAY be safe using pip to install something that isn't available in your package manager, but it's possible to run into dependency versioning conflicts. Be careful. But installing into your default Python, if it's not the system Python, is absolutely safe. Or rather, if it breaks anything, then it's not your fault - it's the fault of something depending on the system Python but not using an absolute shebang :) ChrisA From tim at akwebsoft.com Sun Dec 1 19:41:28 2019 From: tim at akwebsoft.com (Tim Johnson) Date: Sun, 1 Dec 2019 15:41:28 -0900 Subject: ModuleNotFoundError with click module In-Reply-To: References: <32cc3ffc-bb43-284f-dfd8-7451ba100ec7@akwebsoft.com> Message-ID: <49e16310-3ef8-acf2-9583-6843a81f1bd6@akwebsoft.com> On 12/1/19 12:26 AM, Peter Otten wrote: > Tim Johnson wrote: > >> Using linux ubuntu 16.04 with bash shell. >> Am retired python programmer, but not terribly current. >> I have moderate bash experience. >> >> When trying to install pgadmin4 via apt I get the following error >> traceback when pgadmin4 is invoked: >> >> Traceback (most recent call last): >> File "setup.py", line 17, in >> from pgadmin.model import db, User, Version, ServerGroup, Server, \ >> File "/usr/share/pgadmin4/web/pgadmin/__init__.py", line 19, in >> from flask import Flask, abort, request, current_app, session, url_for >> File "/usr/local/lib/python3.7/site-packages/flask/__init__.py", line >> 21, in >> from .app import Flask >> File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 34, >> in >> from . import cli >> File "/usr/local/lib/python3.7/site-packages/flask/cli.py", line 25, in >> >> import click >> ModuleNotFoundError: No module named 'click' >> >> >> If I invoke python3 (/usr/local/bin/python3), version 3.7.2 and invoke >> >>> import click >> click is imported successfully. >> >> In this invocation, sys.path is: >> ['', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', >> '/usr/local/lib/python3.7/lib-dynload', >> '/home/tim/.local/lib/python3.7/site-packages', >> '/usr/local/lib/python3.7/site-packages'] >> >> $PYTHONPATH is empty when the bash shell is invoked >> >> $PATH as follows: >> > /home/tim/bin:/home/tim/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin >> click.py can be found at >> /usr/local/lib/python3.7/site-packages/pipenv/patched/piptools/ >> in turn click.py imports click, presumably as the package, >> which appears to be at >> /usr/local/lib/python3.7/site-packages/pipenv/vendor/click >> >> Any number of settings of PYTHONPATH to the various paths above has >> failed to resolve the ModuleNotFoundError >> Same issues with attempting install from a virtual environment. >> >> Any help will be appreciated. >> thanks >> tim >> > I'm too lazy to look into the details of your paths -- I'd just make sure > that click is installed with the same interpreter and user as pgadmin4, e. > g. globally > > $ sudo /usr/local/bin/python3 -m pip install click > $ sudo /usr/local/bin/python3 path/to/setup.py install # or whatever it > takes to install pgadmin4 OK. Now I have /usr/local/lib/python3.7/site-packages/Click-7.0.dist-info/ which holds the following files: INSTALLER? LICENSE.txt? METADATA? RECORD? top_level.txt? WHEEL I haven't a clue as to how to proceed! Never seen this before ... Furthermore, google is offering me nothing conclusive. Where to go from here! -- Tim tj49.com From tim at akwebsoft.com Sun Dec 1 20:19:45 2019 From: tim at akwebsoft.com (Tim Johnson) Date: Sun, 1 Dec 2019 16:19:45 -0900 Subject: ModuleNotFoundError with click module In-Reply-To: <49e16310-3ef8-acf2-9583-6843a81f1bd6@akwebsoft.com> References: <32cc3ffc-bb43-284f-dfd8-7451ba100ec7@akwebsoft.com> <49e16310-3ef8-acf2-9583-6843a81f1bd6@akwebsoft.com> Message-ID: <510be00b-d87c-8054-cb5d-efb9179bd43b@akwebsoft.com> On 12/1/19 3:41 PM, Tim Johnson wrote: > > On 12/1/19 12:26 AM, Peter Otten wrote: >> Tim Johnson wrote: >> >>> Using linux ubuntu 16.04 with bash shell. >>> Am retired python programmer, but not terribly current. >>> I have moderate bash experience. >>> >>> When trying to install pgadmin4 via apt I get the following error >>> traceback when pgadmin4 is invoked: >>> >>> Traceback (most recent call last): >>> ? File "setup.py", line 17, in >>> ? from pgadmin.model import db, User, Version, ServerGroup, Server, \ >>> ? File "/usr/share/pgadmin4/web/pgadmin/__init__.py", line 19, in >>> >>> ? from flask import Flask, abort, request, current_app, session, >>> url_for >>> ? File "/usr/local/lib/python3.7/site-packages/flask/__init__.py", line >>> 21, in >>> ? from .app import Flask >>> ? File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 34, >>> in >>> ? from . import cli >>> File "/usr/local/lib/python3.7/site-packages/flask/cli.py", line 25, in >>> >>> import click >>> ModuleNotFoundError: No module named 'click' >>> >>> >>> If I invoke python3 (/usr/local/bin/python3), version 3.7.2 and invoke >>> ? >>> import click >>> click is imported successfully. >>> >>> In this invocation, sys.path is: >>> ['', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', >>> '/usr/local/lib/python3.7/lib-dynload', >>> '/home/tim/.local/lib/python3.7/site-packages', >>> '/usr/local/lib/python3.7/site-packages'] >>> >>> $PYTHONPATH is empty when the bash shell is invoked >>> >>> $PATH as follows: >>> >> /home/tim/bin:/home/tim/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin >> >>> click.py can be found at >>> /usr/local/lib/python3.7/site-packages/pipenv/patched/piptools/ >>> in turn click.py imports click, presumably as the package, >>> which appears to be at >>> /usr/local/lib/python3.7/site-packages/pipenv/vendor/click >>> >>> Any number of settings of PYTHONPATH to the various paths above has >>> failed to resolve the ModuleNotFoundError >>> Same issues with attempting install from a virtual environment. >>> >>> Any help will be appreciated. >>> thanks >>> tim >>> >> I'm too lazy to look into the details of your paths -- I'd just make >> sure >> that click is installed with the same interpreter and user as >> pgadmin4, e. >> g. globally >> >> $ sudo /usr/local/bin/python3 -m pip install click >> $ sudo /usr/local/bin/python3 path/to/setup.py install? # or whatever it >> takes to install pgadmin4 > > OK. Now I have > > /usr/local/lib/python3.7/site-packages/Click-7.0.dist-info/ > > which holds the following files: > > INSTALLER? LICENSE.txt? METADATA? RECORD? top_level.txt? WHEEL > > I haven't a clue as to how to proceed! Never seen this before ... > > Furthermore, google is offering me nothing conclusive. > > Where to go from here! P.S. It looks like that directory is sort of a stub; regardless of my take on it I am no longer having the ModuleNotFoundError. Peter has a been a great help. Couldn't have done it without him. cheers -- Tim tj49.com From python.list at tim.thechases.com Sun Dec 1 21:50:16 2019 From: python.list at tim.thechases.com (Tim Chase) Date: Sun, 1 Dec 2019 20:50:16 -0600 Subject: increasing the page size of a dbm store? In-Reply-To: <0EAFC9E0-02D2-4745-99B7-654B7A445EDD@barrys-emacs.org> References: <20191126192432.535ee241@bigbox.attlocal.net> <0EAFC9E0-02D2-4745-99B7-654B7A445EDD@barrys-emacs.org> Message-ID: <20191201205016.577673f9@bigbox.attlocal.net> > Maybe port to SQLite? I would not choose dbm these days. After sparring with it a while, I tweaked the existing job so that it chunked things into dbm-appropriate sizes to limp through; for the subsequent job (where I would have used dbm again) I went ahead and switched to sqlite and had no further issues. I'm not sure if it's worth mentioning the issue in the docs for the dbm module so others don't bump against it. I'm not sure if the limit is on sum(size(key) for key in db) or the number of keys total. Just not the sort of thing I'd want someone to be depending on, unaware of the potential pitfalls. Thanks, -tkc From niktnobodynikt at gmail.com Mon Dec 2 02:52:44 2019 From: niktnobodynikt at gmail.com (niktnobodynikt at gmail.com) Date: Sun, 1 Dec 2019 23:52:44 -0800 (PST) Subject: "Don't install on the system Python" In-Reply-To: <0a065288-f851-4afc-9104-f5b5ca5188f9@googlegroups.com> References: <0a065288-f851-4afc-9104-f5b5ca5188f9@googlegroups.com> Message-ID: W dniu niedziela, 1 grudnia 2019 05:42:35 UTC+1 u?ytkownik John Ladasky napisa?: > For years, I've read warnings about not installing one's personal stack of Python modules on top of the system Python. It is possible to corrupt the OS, or so I've gathered. > > Well, I've never heeded this advice, and so far nothing bad has happened to me. I don't like Anaconda, or virtual environments in general. I don't like heavyweight IDE's. I like to be able to type "python3" at the command prompt and be sure what I'll be getting. I have multiple user accounts on a system that I manage, and I want every user account to have access to the same modules. > > Maybe the modules that I require are safe to install on the system Python, I'm not sure. My must-haves are mostly scientific computing and data management modules: Numpy, Scipy, Scikit-learn, Matplotlib, Pandas, Biopython, and Tensorflow. I also use PyQt5 from time to time. > > Can anyone provide concrete examples of problems arising from installing modules on top of the system Python? Am I courting disaster? I did not heard of such problems. But I have another warning. Ubuntu 18.4 uses Python 3.6. Do not try to install 3.7 or 3.8 as systemwide python3 version. It breaks some programs including standard terminal. From rosuav at gmail.com Mon Dec 2 03:15:58 2019 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 2 Dec 2019 19:15:58 +1100 Subject: "Don't install on the system Python" In-Reply-To: References: <0a065288-f851-4afc-9104-f5b5ca5188f9@googlegroups.com> Message-ID: On Mon, Dec 2, 2019 at 6:56 PM wrote: > > W dniu niedziela, 1 grudnia 2019 05:42:35 UTC+1 u?ytkownik John Ladasky napisa?: > > > For years, I've read warnings about not installing one's personal stack of Python modules on top of the system Python. It is possible to corrupt the OS, or so I've gathered. > > > > Well, I've never heeded this advice, and so far nothing bad has happened to me. I don't like Anaconda, or virtual environments in general. I don't like heavyweight IDE's. I like to be able to type "python3" at the command prompt and be sure what I'll be getting. I have multiple user accounts on a system that I manage, and I want every user account to have access to the same modules. > > > > Maybe the modules that I require are safe to install on the system Python, I'm not sure. My must-haves are mostly scientific computing and data management modules: Numpy, Scipy, Scikit-learn, Matplotlib, Pandas, Biopython, and Tensorflow. I also use PyQt5 from time to time. > > > > Can anyone provide concrete examples of problems arising from installing modules on top of the system Python? Am I courting disaster? > > > I did not heard of such problems. But I have another warning. Ubuntu 18.4 uses Python 3.6. Do not try to install 3.7 or 3.8 as systemwide python3 version. It breaks some programs including standard terminal. > And that's where the distinction between "system Python" and "default Python" comes in. You are absolutely right that you shouldn't replace the *system* Python. However, changing what the command "python3" runs is safe. ChrisA From __peter__ at web.de Mon Dec 2 03:46:37 2019 From: __peter__ at web.de (Peter Otten) Date: Mon, 02 Dec 2019 09:46:37 +0100 Subject: ModuleNotFoundError with click module References: <32cc3ffc-bb43-284f-dfd8-7451ba100ec7@akwebsoft.com> <49e16310-3ef8-acf2-9583-6843a81f1bd6@akwebsoft.com> <510be00b-d87c-8054-cb5d-efb9179bd43b@akwebsoft.com> Message-ID: Tim Johnson wrote: >> OK. Now I have >> >> /usr/local/lib/python3.7/site-packages/Click-7.0.dist-info/ >> >> which holds the following files: >> >> INSTALLER LICENSE.txt METADATA RECORD top_level.txt WHEEL >> >> I haven't a clue as to how to proceed! Never seen this before ... Just leave it alone ;) >> Furthermore, google is offering me nothing conclusive. >> >> Where to go from here! > > P.S. It looks like that directory is sort of a stub; regardless of my > take on it I am no longer having the ModuleNotFoundError. Once you can import it you can find the actual module or package with $ /usr/bin/python3.7 -c 'import click; print(click.__file__)' In this case it's a package, so you'll probably see (something like) /usr/local/lib/python3.7/site-packages/click/__init__.py rather than /usr/local/lib/python3.7/site-packages/click.py /usr/local/lib/python3.7/site-packages/ From soyeomul at doraji.xyz Mon Dec 2 03:58:42 2019 From: soyeomul at doraji.xyz (=?utf-8?B?7Zmp67OR7Z2s?=) Date: Mon, 02 Dec 2019 17:58:42 +0900 Subject: tab replace to space 4 References: Message-ID: Hi, Gilmeh^^^ > We are Python people, aren't we? Looks good, i did copy it [1], and thanks^^^ [1] https://gitlab.com/soyeomul/test/blob/master/untabify.py Sincerely, -- ^????? _????_ ?????_^))// From aishan0403 at gmail.com Mon Dec 2 10:40:41 2019 From: aishan0403 at gmail.com (A S) Date: Mon, 2 Dec 2019 07:40:41 -0800 (PST) Subject: Extract sentences in nested parentheses using Python Message-ID: <7a365fa0-e721-4a95-8c30-d3661301a7b2@googlegroups.com> I am trying to extract all strings in nested parentheses (along with the parentheses itself) in my .txt file. Please see the sample .txt file that I have used in this example here: (https://drive.google.com/open?id=1UKc0ZgY9Fsz5O1rSeBCLqt5dwZkMaQgr). I have tried and done up three different codes but none of them seems to be able to extract all the nested parentheses. They can only extract a portion of the nested parentheses. Any advice on what I've done wrong could really help! Here are the three codes I have done so far: 1st attempt: import re from os.path import join def balanced_braces(args): parts = [] for arg in args: if '(' not in arg: continue chars = [] n = 0 for c in arg: if c == '(': if n > 0: chars.append(c) n += 1 elif c == ')': n -= 1 if n > 0: chars.append(c) elif n == 0: parts.append(''.join(chars).lstrip().rstrip()) chars = [] elif n > 0: chars.append(c) return parts with open('lan sample text file.txt','r') as fd: #for words in fd.readlines(): t1 = balanced_braces(fd); print(t1) Output: ['"xE\'", PUT(xx.xxxx.),"\'"', '"TRUuuuth"', "xxx(xx_ix as format 'xxxx-xx') lec &jgjsd_vnv.", '"xE\'", PUT(xx.xxxx.),"\'"', '"CUuuiiiiuth"', "xxx(xx_ix as format 'xxxx-xx') lec &jgjsd_vnv."] 2nd attempt: from pyparsing import nestedExpr matchedParens = nestedExpr('(',')') with open('lan sample text file.txt','r') as fd: for words in fd.readlines(): for e in matchedParens.searchString(words): print(e) Output: [['"xE\'"', ',', 'PUT', ['xx.xxxx.'], ',', '"\'"']] [['"TRUuuuth"']] [['xxx', ['xx_ix', 'as', 'format', "'xxxx-xx'"], 'gff', '&jfjfsj_jfjfj.']] [['xxx', ['xx_ix', 'as', 'format', "'xxxx-xx'"], 'lec', '&jgjsd_vnv.']] [['"xE\'"', ',', 'PUT', ['xx.xxxx.'], ',', '"\'"']] [['"CUuuiiiiuth"']] [['xxx', ['xx_ix', 'as', 'format', "'xxxx-xx'"], 'gff', '&jfjfsj_jfjfj.']] [['xxx', ['xx_ix', 'as', 'format', "'xxxx-xx'"], 'lec', '&jgjsd_vnv.']] 3rd attempt: def parse_segments(source, recurse=False): unmatched_count = 0 start_pos = 0 opened = False open_pos = 0 cur_pos = 0 finished = [] segments = [] for character in source: #scan for mismatched parenthesis: if character == '(': unmatched_count += 1 if not opened: open_pos = cur_pos opened = True if character == ')': unmatched_count -= 1 if opened and unmatched_count == 0: segment = source[open_pos:cur_pos+1] segments.append(segment) clean = source[start_pos:open_pos] if clean: finished.append(clean) opened = False start_pos = cur_pos+1 cur_pos += 1 # assert unmatched_count == 0 if start_pos != cur_pos: #get anything that was left over here finished.append(source[start_pos:cur_pos]) #now check on recursion: for item in segments: #get rid of bounding parentheses: pruned = item[1:-1] if recurse: results = parse_tags(pruned, recurse) finished.expand(results) else: finished.append(pruned) return finished with open('lan sample text file.txt','r') as fd: for words in fd.readlines(): t = parse_segments(words) print(t) Output: ['kkkkk;\n'] ['\n'] [' select xx', ' jdfjhf:jhfjj from xxxx_x_xx_L ;\n', '"xE\'", PUT(xx.xxxx.),"\'"'] ['quit; \n'] ['\n'] ['/* 1.xxxxx FROM xxxx_x_Ex_x */ \n'] ['proc sql; ', ';\n', '"TRUuuuth"'] ['hhhjhfjs as fdsjfsj:\n'] ['select * from djfkjd to jfkjs\n'] ['(\n'] ['SELECT abc AS abc1, abc_2_ AS efg, abc_fg, fkdkfj_vv, jjsflkl_ff, fjkdsf_jfkj\n'] ['\tFROM &xxx..xxx_xxx_xxE\n'] ["where ((xxx(xx_ix as format 'xxxx-xx') gff &jfjfsj_jfjfj.) and \n"] [' ', ')\n', "xxx(xx_ix as format 'xxxx-xx') lec &jgjsd_vnv."] [' );\n'] ['\n'] ['\n'] ['jjjjjj;\n'] ['\n'] [' select xx', ' jdfjhf:jhfjj from xxxx_x_xx_L ;\n', '"xE\'", PUT(xx.xxxx.),"\'"'] ['quit; \n'] ['\n'] ['/* 1.xxxxx FROM xxxx_x_Ex_x */ \n'] ['proc sql; ', ';\n', '"CUuuiiiiuth"'] ['hhhjhfjs as fdsjfsj:\n'] ['select * from djfkjd to jfkjs\n'] ['(SELECT abc AS abc1, abc_2_ AS efg, abc_fg, fkdkfj_vv, jjsflkl_ff, fjkdsf_jfkj\n'] ['\tFROM &xxx..xxx_xxx_xxE\n'] ["where ((xxx(xx_ix as format 'xxxx-xx') gff &jfjfsj_jfjfj.) and \n"] [' ', ')\n', "xxx(xx_ix as format 'xxxx-xx') lec &jgjsd_vnv."] [' );'] My intended Output that I am unable to get should look something like this: ("xE'", PUT(xx.xxxx.),"'") ("TRUuuuth") ( SELECT abc AS abc1, abc_2_ AS efg, abc_fg, fkdkfj_vv, jjsflkl_ff, fjkdsf_jfkj FROM &xxx..xxx_xxx_xxE where ((xxx(xx_ix as format 'xxxx-xx') gff &jfjfsj_jfjfj.) and (xxx(xx_ix as format 'xxxx-xx') lec &jgjsd_vnv.)) ) ("xE'", PUT(xx.xxxx.),"'") ("CUuuiiiiuth") (SELECT abc AS abc1, abc_2_ AS efg, abc_fg, fkdkfj_vv, jjsflkl_ff, fjkdsf_jfkj FROM &xxx..xxx_xxx_xxE where ((xxx(xx_ix as format 'xxxx-xx') gff &jfjfsj_jfjfj.) and (xxx(xx_ix as format 'xxxx-xx') lec &jgjsd_vnv.))(( )) ) From __peter__ at web.de Mon Dec 2 12:00:50 2019 From: __peter__ at web.de (Peter Otten) Date: Mon, 02 Dec 2019 18:00:50 +0100 Subject: Extract sentences in nested parentheses using Python References: <7a365fa0-e721-4a95-8c30-d3661301a7b2@googlegroups.com> Message-ID: A S wrote: I think I've seen this question before ;) > I am trying to extract all strings in nested parentheses (along with the > parentheses itself) in my .txt file. Please see the sample .txt file that > I have used in this example here: > (https://drive.google.com/open?id=1UKc0ZgY9Fsz5O1rSeBCLqt5dwZkMaQgr). > > I have tried and done up three different codes but none of them seems to > be able to extract all the nested parentheses. They can only extract a > portion of the nested parentheses. Any advice on what I've done wrong > could really help! > > Here are the three codes I have done so far: > > 1st attempt: > > import re > from os.path import join > > def balanced_braces(args): > parts = [] > for arg in args: > if '(' not in arg: > continue There could still be a ")" that you miss > chars = [] > n = 0 > for c in arg: > if c == '(': > if n > 0: > chars.append(c) > n += 1 > elif c == ')': > n -= 1 > if n > 0: > chars.append(c) > elif n == 0: > parts.append(''.join(chars).lstrip().rstrip()) > chars = [] > elif n > 0: > chars.append(c) > return parts It's probably easier to understand and implement when you process the complete text at once. Then arbitrary splits don't get in the way of your quest for ( and ). You just have to remember the position of the first opening ( and number of opening parens that have to be closed before you take the complete expression: level: 00011112222100 text: abc(def(gh))ij when we are here^ we need^ A tentative implementation: $ cat parse.py import re NOT_SET = object() def scan(text): level = 0 start = NOT_SET for m in re.compile("[()]").finditer(text): if m.group() == ")": level -= 1 if level < 0: raise ValueError("underflow: more closing than opening parens") if level == 0: # outermost closing parenthesis: # deliver enclosed string including parens. yield text[start:m.end()] start = NOT_SET elif m.group() == "(": if level == 0: # outermost opening parenthesis: remember position. assert start is NOT_SET start = m.start() level += 1 else: assert False if level > 0: raise ValueError("unclosed parens remain") if __name__ == "__main__": with open("lan sample text file.txt") as instream: text = instream.read() for chunk in scan(text): print(chunk) $ python3 parse.py ("xE'", PUT(xx.xxxx.),"'") ("TRUuuuth") From Chris.Clark at actian.com Mon Dec 2 12:26:00 2019 From: Chris.Clark at actian.com (Chris Clark) Date: Mon, 2 Dec 2019 17:26:00 +0000 Subject: array and struct 64-bit Linux change in behavior Python 3.7 and 2.7 Message-ID: Test case: import array array.array('L', [0]) # x.itemsize == 8 rather than 4 This works fine (returns 4) under Windows Python 3.7.3 64-bit build. Under Ubuntu; Python 2.7.15rc1, 3.6.5, 3.70b3 64-bit this returns 8. Documentation at https://docs.python.org/3/library/array.html explicitly states 'L' is for size 4. It impacts all uses types of array (e.g. reading from byte strings). The struct module is a little different: import struct x = struct.pack('L', 0) # len(x) ===8 rather than 4 This can be worked around by using '=L' - which is not well documented - so this maybe a doc issue. Wanted to post here for comments before opening a bug at https://bugs.python.org/ Is anyone seeing this under Debian/Ubuntu? Chris From Richard at Damon-family.org Mon Dec 2 12:47:06 2019 From: Richard at Damon-family.org (Richard Damon) Date: Mon, 2 Dec 2019 12:47:06 -0500 Subject: array and struct 64-bit Linux change in behavior Python 3.7 and 2.7 In-Reply-To: References: Message-ID: <0BD00D4B-5445-46F8-BDE8-168939332F63@Damon-family.org> On Dec 2, 2019, at 12:32 PM, Chris Clark wrote: > > ?Test case: > > import array > array.array('L', [0]) > # x.itemsize == 8 rather than 4 > > This works fine (returns 4) under Windows Python 3.7.3 64-bit build. > > Under Ubuntu; Python 2.7.15rc1, 3.6.5, 3.70b3 64-bit this returns 8. Documentation at https://docs.python.org/3/library/array.html explicitly states 'L' is for size 4. > It impacts all uses types of array (e.g. reading from byte strings). > > The struct module is a little different: > > import struct > x = struct.pack('L', 0) > # len(x) ===8 rather than 4 > > This can be worked around by using '=L' - which is not well documented - so this maybe a doc issue. > > Wanted to post here for comments before opening a bug at https://bugs.python.org/ > > Is anyone seeing this under Debian/Ubuntu? > > > Chris > Documentation that I see says *Minimum* size is 4, nothing says that it will be 4 I wouldn?t be surprized if ?I? gave you a size of 4 on that platform (and maybe even on many 32 bit platforms too) From rgaddi at highlandtechnology.invalid Mon Dec 2 12:55:16 2019 From: rgaddi at highlandtechnology.invalid (Rob Gaddi) Date: Mon, 2 Dec 2019 09:55:16 -0800 Subject: array and struct 64-bit Linux change in behavior Python 3.7 and 2.7 In-Reply-To: References: Message-ID: On 12/2/19 9:26 AM, Chris Clark wrote: > Test case: > > import array > array.array('L', [0]) > # x.itemsize == 8 rather than 4 > > This works fine (returns 4) under Windows Python 3.7.3 64-bit build. > > Under Ubuntu; Python 2.7.15rc1, 3.6.5, 3.70b3 64-bit this returns 8. Documentation at https://docs.python.org/3/library/array.html explicitly states 'L' is for size 4. > It impacts all uses types of array (e.g. reading from byte strings). > > The struct module is a little different: > > import struct > x = struct.pack('L', 0) > # len(x) ===8 rather than 4 > > This can be worked around by using '=L' - which is not well documented - so this maybe a doc issue. > > Wanted to post here for comments before opening a bug at https://bugs.python.org/ > > Is anyone seeing this under Debian/Ubuntu? > > > Chris > I'd say not a bug, at least in array. Reading that array documentation you linked, 4 is explicitly the MINIMUM size in bytes, not the guaranteed size. The struct situation is, as you said, a bit different. I believe that with the default native alignment @, you're seeing 4-byte data padded to an 8-byte alignment, not 8-byte data. That does seem to go against what the struct documentation says, "Padding is only automatically added between successive structure members. No padding is added at the beginning or the end of the encoded struct." = alignment is documented as having the platform native byte-order, but the size and alignment is standardized as having no padding, which is exactly the behavior you seem to want. The documentation is a bit obtuse and scattered, but no more than any other. From tim at akwebsoft.com Mon Dec 2 14:48:39 2019 From: tim at akwebsoft.com (Tim Johnson) Date: Mon, 2 Dec 2019 10:48:39 -0900 Subject: ModuleNotFoundError with click module In-Reply-To: References: <32cc3ffc-bb43-284f-dfd8-7451ba100ec7@akwebsoft.com> <49e16310-3ef8-acf2-9583-6843a81f1bd6@akwebsoft.com> <510be00b-d87c-8054-cb5d-efb9179bd43b@akwebsoft.com> Message-ID: On 12/1/19 11:46 PM, Peter Otten wrote: > Tim Johnson wrote: > >>> OK. Now I have >>> >>> /usr/local/lib/python3.7/site-packages/Click-7.0.dist-info/ >>> >>> which holds the following files: >>> >>> INSTALLER LICENSE.txt METADATA RECORD top_level.txt WHEEL >>> >>> I haven't a clue as to how to proceed! Never seen this before ... > Just leave it alone ;) > >>> Furthermore, google is offering me nothing conclusive. >>> >>> Where to go from here! >> P.S. It looks like that directory is sort of a stub; regardless of my >> take on it I am no longer having the ModuleNotFoundError. > Once you can import it you can find the actual module or package with > > $ /usr/bin/python3.7 -c 'import click; print(click.__file__)' > > In this case it's a package, so you'll probably see (something like) > > /usr/local/lib/python3.7/site-packages/click/__init__.py > > rather than > > /usr/local/lib/python3.7/site-packages/click.py > /usr/local/lib/python3.7/site-packages/ Good Explanation. Thanks again Peter. new since python 1.5 -- Tim tj49.com From barry at barrys-emacs.org Mon Dec 2 16:25:05 2019 From: barry at barrys-emacs.org (Barry Scott) Date: Mon, 2 Dec 2019 21:25:05 +0000 Subject: array and struct 64-bit Linux change in behavior Python 3.7 and 2.7 In-Reply-To: References: Message-ID: <475624D9-15D8-4A82-A8B4-5A174BA87DCD@barrys-emacs.org> > On 2 Dec 2019, at 17:55, Rob Gaddi wrote: > > On 12/2/19 9:26 AM, Chris Clark wrote: >> Test case: >> import array >> array.array('L', [0]) >> # x.itemsize == 8 rather than 4 >> This works fine (returns 4) under Windows Python 3.7.3 64-bit build. >> Under Ubuntu; Python 2.7.15rc1, 3.6.5, 3.70b3 64-bit this returns 8. Documentation at https://docs.python.org/3/library/array.html explicitly states 'L' is for size 4. >> It impacts all uses types of array (e.g. reading from byte strings). >> The struct module is a little different: >> import struct >> x = struct.pack('L', 0) >> # len(x) ===8 rather than 4 >> This can be worked around by using '=L' - which is not well documented - so this maybe a doc issue. >> Wanted to post here for comments before opening a bug at https://bugs.python.org/ >> Is anyone seeing this under Debian/Ubuntu? >> Chris > > I'd say not a bug, at least in array. Reading that array documentation you linked, 4 is explicitly the MINIMUM size in bytes, not the guaranteed size. I'm wondering how useful it is that for array you can read from a file but have no ideas how many bytes each item needs. If I have a file with int32_t in it I cannot from the docs know how to read that file into an array. > > The struct situation is, as you said, a bit different. I believe that with the default native alignment @, you're seeing 4-byte data padded to an 8-byte alignment, not 8-byte data. That does seem to go against what the struct documentation says, "Padding is only automatically added between successive structure members. No padding is added at the beginning or the end of the encoded struct." The 'L' in struct is documented for 3.7 to use 4 bytes, but in fact uses 8, on fedora 31. Doc bug? >>> x=struct.pack('L',0x102030405) >>> x b'\x05\x04\x03\x02\x01\x00\x00\x00' Given I have exact control with b, h, i, and q but L is not fixed in size I'm not sure how it can be used with certainty across OS and versions. Barry > > = alignment is documented as having the platform native byte-order, but the size and alignment is standardized as having no padding, which is exactly the behavior you seem to want. The documentation is a bit obtuse and scattered, but no more than any other. > -- > https://mail.python.org/mailman/listinfo/python-list From post at tinita.de Mon Dec 2 17:04:21 2019 From: post at tinita.de (=?ISO-8859-15?Q?Tina_M=FCller?=) Date: Mon, 2 Dec 2019 23:04:21 +0100 (CET) Subject: [ANN] PyYAML-5.2: YAML parser and emitter for Python Message-ID: ======================= Announcing PyYAML-5.2 ======================= A new release of PyYAML is now available: https://pypi.org/project/PyYAML/ This fixes some incompatibilities introduced in version 5.1 and also removes another possibility of loading arbitrary code. Changes ======= * Repair incompatibilities introduced with 5.1. The default Loader was changed, but several methods like add_constructor still used the old default https://github.com/yaml/pyyaml/pull/279 -- A more flexible fix for custom tag constructors https://github.com/yaml/pyyaml/pull/287 -- Change default loader for yaml.add_constructor https://github.com/yaml/pyyaml/pull/305 -- Change default loader for add_implicit_resolver, add_path_resolver * Make FullLoader safer by removing python/object/apply from the default FullLoader https://github.com/yaml/pyyaml/pull/347 -- Move constructor for object/apply to UnsafeConstructor * Fix bug introduced in 5.1 where quoting went wrong on systems with sys.maxunicode <= 0xffff https://github.com/yaml/pyyaml/pull/276 -- Fix logic for quoting special characters * Other PRs: https://github.com/yaml/pyyaml/pull/280 -- Update CHANGES for 5.1 Resources ========= PyYAML IRC Channel: #pyyaml on irc.freenode.net PyYAML homepage: https://github.com/yaml/pyyaml PyYAML documentation: http://pyyaml.org/wiki/PyYAMLDocumentation Source and binary installers: https://pypi.org/project/PyYAML/ GitHub repository: https://github.com/yaml/pyyaml/ Bug tracking: https://github.com/yaml/pyyaml/issues YAML homepage: http://yaml.org/ YAML-core mailing list: http://lists.sourceforge.net/lists/listinfo/yaml-core About PyYAML ============ YAML is a data serialization format designed for human readability and interaction with scripting languages. PyYAML is a YAML parser and emitter for Python. PyYAML features a complete YAML 1.1 parser, Unicode support, pickle support, capable extension API, and sensible error messages. PyYAML supports standard YAML tags and provides Python-specific tags that allow to represent an arbitrary Python object. PyYAML is applicable for a broad range of tasks from complex configuration files to object serialization and persistence. Example ======= >>> import yaml >>> yaml.full_load(""" ... name: PyYAML ... description: YAML parser and emitter for Python ... homepage: https://github.com/yaml/pyyaml ... keywords: [YAML, serialization, configuration, persistence, pickle] ... """) {'keywords': ['YAML', 'serialization', 'configuration', 'persistence', 'pickle'], 'homepage': 'https://github.com/yaml/pyyaml', 'description': 'YAML parser and emitter for Python', 'name': 'PyYAML'} >>> print(yaml.dump(_)) name: PyYAML homepage: https://github.com/yaml/pyyaml description: YAML parser and emitter for Python keywords: [YAML, serialization, configuration, persistence, pickle] Maintainers =========== The following people are currently responsible for maintaining PyYAML: * Ingy d?t Net * Tina Mueller * Matt Davis and many thanks to all who have contribributed! See: https://github.com/yaml/pyyaml/pulls Copyright ========= Copyright (c) 2017-2019 Ingy d?t Net Copyright (c) 2006-2016 Kirill Simonov The PyYAML module was written by Kirill Simonov . It is currently maintained by the YAML and Python communities. PyYAML is released under the MIT license. See the file LICENSE for more details. From PythonList at DancesWithMice.info Mon Dec 2 17:22:19 2019 From: PythonList at DancesWithMice.info (DL Neil) Date: Tue, 3 Dec 2019 11:22:19 +1300 Subject: Extract sentences in nested parentheses using Python In-Reply-To: References: <7a365fa0-e721-4a95-8c30-d3661301a7b2@googlegroups.com> Message-ID: <1d689634-c3db-dd3f-c432-111604d5a784@DancesWithMice.info> On 3/12/19 6:00 AM, Peter Otten wrote: > A S wrote: > I think I've seen this question before ;) In addition to 'other reasons' for @Peter's comment, it is a common ComSc worked-problem or assignment. (in which case, we'd appreciate being told that you/OP is asking for help with "homework") >> I am trying to extract all strings in nested parentheses (along with the >> parentheses itself) in my .txt file. Please see the sample .txt file that >> I have used in this example here: >> (https://drive.google.com/open?id=1UKc0ZgY9Fsz5O1rSeBCLqt5dwZkMaQgr). >> >> I have tried and done up three different codes but none of them seems to >> be able to extract all the nested parentheses. They can only extract a >> portion of the nested parentheses. Any advice on what I've done wrong >> could really help! One approach is to research in the hope that there are already existing tools or techniques which may help/save you from 'reinventing the wheel' - when you think about it, a re-statement of open-source objectives. How does the Python interpreter break-down Python (text) code into its constituent parts ("tokens") *including* parentheses? Such are made available in (perhaps) a lesser-known corner of the PSL (Python Standard Library). Might you be able to use one such tool? The ComSc technique which sprang to mind involves "stacks" (a LIFO data structure) and "RPN" (Reverse Polish Notation). Whereas we like people to take their turn when it comes to limited resources, eg to form a "queue" to purchase/pay for goods at the local store, which is "FIFO" (first-in, first-out); a "stack"/LIFO (last-in, first-out) can be problematic to put into practical application. There are plenty of Python implementations or you can 'roll your own' with a list. Again, I'd likely employ a "deque" from the PSL's Collections library (although as a "stack" rather than as a "double-ended queue"), because the optimisation comes "free". (to my laziness, but after some kind soul sweated-bullets to make it fast (in both senses) for 'the rest of us'!) > It's probably easier to understand and implement when you process the > complete text at once. Then arbitrary splits don't get in the way of your > quest for ( and ). You just have to remember the position of the first > opening ( and number of opening parens that have to be closed before you > take the complete expression: +1 but as a 'silver surfer', I don't like to be asked to "remember" anything! > level: 00011112222100 > we need^ Consider: original_text (the contents of the .txt file - add buffering if volumes are huge) current_text (the characters we have processed/"recognised" so-far) stack (what an original name for such a data-structure! Which will contain each of the partial parenthetical expressions found - but yet to be proven/made complete) set current_text to NULSTRING for each current_character in original_text: if current_character is LEFT_PARENTHESIS: push current_text to stack set current_text to LEFT_PARENTHESIS concatenate current_character with current_text if current_character is RIGHT_PARENTHESIS: # current_text is a parenthetical expression # do with it what you will pop the stack set current_text to the ex-stack string \ concat current_text's p-expn Once working: cover 'special cases' (after above loop), eg original_text which doesn't begin and/or end with parentheses; and error cases, eg pop-ping a NULSTRING, or thinking things are finished but the stack is not yet empty - likely events from unbalanced parentheses! original text = "abc(def(gh))ij" event 1: in-turn, concatenate characters "abc" as current_text event 2: locate (first) left-parenthesis, push current_text to stack(&) event 3: concatenate "(def" event 4: push, likewise event 5: concatenate "(gh" event 6: locate (first) right-parenthesis (matches to left-parenthesis begining the current_string!) result?: ?print current_text? event 7: pop stack and redefine current_text as "(def(gh)" event 8: repeat, per event 6 event 9: current_text will now become "(def(gh))" event 10: (special case) run-out of input at "(def(gh)ij" event 11: (special case) pop (!any) stack and report "abc(def(gh)" NB not sure of need for a "level" number; but if required, you can infer that at any time, from the depth/len() of the stack! NBB being a simple-boy, my preference is for a 'single layer' of code, cf @Peter's generator. Regardless the processes are "tokenisation" and "recognition". At the back of my mind, was the notion that you may (eventually) be required to work with more than parentheses, eg pair-wise square-brackets and/or quotation-marks. In which case, you will need to also 'push' the token and check for token-pairs when 'pop-ping', as well as (perhaps) recognising lists of tokens to tokenise instead of the two parenthesis characters alone. In which case, I'd take a serious look at the Python Language Services rather than taking a 'roll your own' approach! Contrarily, if this spec is 'it', then you might consider optimising the search processes which 'trigger' the two stack operations, by re-working the for-loop and utilising string.find() - prioritising whichever parenthesis is left-most/comes-first - assuming LtR text. (apologies if you have already tried this in one of your previous approaches) Unfortunately, such likely results in 'layers' of code, and a generator might well become the tool-of-choice (I say this before @Peter comes back and (quite deservedly) flays me alive!). WebRefs: Python Language Services: https://docs.python.org/3/library/language.html collections ? Container datatypes: https://docs.python.org/3/library/collections.html See also your ComSc text/reference materials. -- Regards =dn From christian at python.org Mon Dec 2 17:37:45 2019 From: christian at python.org (Christian Heimes) Date: Mon, 2 Dec 2019 23:37:45 +0100 Subject: array and struct 64-bit Linux change in behavior Python 3.7 and 2.7 In-Reply-To: <475624D9-15D8-4A82-A8B4-5A174BA87DCD@barrys-emacs.org> References: <475624D9-15D8-4A82-A8B4-5A174BA87DCD@barrys-emacs.org> Message-ID: <94e9ab79-12dd-e891-be80-7df913d5fdf4@python.org> On 02/12/2019 22.25, Barry Scott wrote: > > >> On 2 Dec 2019, at 17:55, Rob Gaddi wrote: >> >> On 12/2/19 9:26 AM, Chris Clark wrote: >>> Test case: >>> import array >>> array.array('L', [0]) >>> # x.itemsize == 8 rather than 4 >>> This works fine (returns 4) under Windows Python 3.7.3 64-bit build. >>> Under Ubuntu; Python 2.7.15rc1, 3.6.5, 3.70b3 64-bit this returns 8. Documentation at https://docs.python.org/3/library/array.html explicitly states 'L' is for size 4. >>> It impacts all uses types of array (e.g. reading from byte strings). >>> The struct module is a little different: >>> import struct >>> x = struct.pack('L', 0) >>> # len(x) ===8 rather than 4 >>> This can be worked around by using '=L' - which is not well documented - so this maybe a doc issue. >>> Wanted to post here for comments before opening a bug at https://bugs.python.org/ >>> Is anyone seeing this under Debian/Ubuntu? >>> Chris >> >> I'd say not a bug, at least in array. Reading that array documentation you linked, 4 is explicitly the MINIMUM size in bytes, not the guaranteed size. > > I'm wondering how useful it is that for array you can read from a file but have no ideas how many bytes each item needs. > If I have a file with int32_t in it I cannot from the docs know how to read that file into an array. > >> >> The struct situation is, as you said, a bit different. I believe that with the default native alignment @, you're seeing 4-byte data padded to an 8-byte alignment, not 8-byte data. That does seem to go against what the struct documentation says, "Padding is only automatically added between successive structure members. No padding is added at the beginning or the end of the encoded struct." > > The 'L' in struct is documented for 3.7 to use 4 bytes, but in fact uses 8, on fedora 31. Doc bug? The documentation of the struct and array module carefully speak of "minimum size in bytes", "standard size" and "native size". It's easy to miss that it doesn't state just "size" for a reason. A long is not int32_t. The actual size of long and unsigned long depend on the ABI of your platform. The standard defined a long as *at least* 4 bytes. On Windows it's always a 32 bit data type. On POSIX sizeof(long) is usually the same as the size of a pointer, 4 bytes on 32 bit platforms and 8 bytes on 64 bit platforms. https://en.wikipedia.org/wiki/Integer_(computer_science)#Long_integer I agree that the behavior is confusing, even for C programmers. Please feel free to open a ticket and request an improvement of the documentation. Christian From torriem at gmail.com Mon Dec 2 18:49:17 2019 From: torriem at gmail.com (Michael Torrie) Date: Mon, 2 Dec 2019 16:49:17 -0700 Subject: increasing the page size of a dbm store? In-Reply-To: <20191201205016.577673f9@bigbox.attlocal.net> References: <20191126192432.535ee241@bigbox.attlocal.net> <0EAFC9E0-02D2-4745-99B7-654B7A445EDD@barrys-emacs.org> <20191201205016.577673f9@bigbox.attlocal.net> Message-ID: <4dd1ac29-8803-a1bd-3158-0cd90f392c48@gmail.com> On 12/1/19 7:50 PM, Tim Chase wrote: > After sparring with it a while, I tweaked the existing job so that it > chunked things into dbm-appropriate sizes to limp through; for the > subsequent job (where I would have used dbm again) I went ahead and > switched to sqlite and had no further issues. How did you replace a key/value store with a relational database? Is a SQLite database fast enough at this sort of thing that it wasn't really designed for? From Richard at Damon-Family.org Mon Dec 2 20:50:16 2019 From: Richard at Damon-Family.org (Richard Damon) Date: Mon, 2 Dec 2019 20:50:16 -0500 Subject: array and struct 64-bit Linux change in behavior Python 3.7 and 2.7 In-Reply-To: <475624D9-15D8-4A82-A8B4-5A174BA87DCD@barrys-emacs.org> References: <475624D9-15D8-4A82-A8B4-5A174BA87DCD@barrys-emacs.org> Message-ID: <2b3d0c1a-aab4-9084-1011-975f911352d8@Damon-Family.org> On 12/2/19 4:25 PM, Barry Scott wrote: > >> On 2 Dec 2019, at 17:55, Rob Gaddi wrote: >> >> On 12/2/19 9:26 AM, Chris Clark wrote: >>> Test case: >>> import array >>> array.array('L', [0]) >>> # x.itemsize == 8 rather than 4 >>> This works fine (returns 4) under Windows Python 3.7.3 64-bit build. >>> Under Ubuntu; Python 2.7.15rc1, 3.6.5, 3.70b3 64-bit this returns 8. Documentation at https://docs.python.org/3/library/array.html explicitly states 'L' is for size 4. >>> It impacts all uses types of array (e.g. reading from byte strings). >>> The struct module is a little different: >>> import struct >>> x = struct.pack('L', 0) >>> # len(x) ===8 rather than 4 >>> This can be worked around by using '=L' - which is not well documented - so this maybe a doc issue. >>> Wanted to post here for comments before opening a bug at https://bugs.python.org/ >>> Is anyone seeing this under Debian/Ubuntu? >>> Chris >> I'd say not a bug, at least in array. Reading that array documentation you linked, 4 is explicitly the MINIMUM size in bytes, not the guaranteed size. > I'm wondering how useful it is that for array you can read from a file but have no ideas how many bytes each item needs. > If I have a file with int32_t in it I cannot from the docs know how to read that file into an array. > >> The struct situation is, as you said, a bit different. I believe that with the default native alignment @, you're seeing 4-byte data padded to an 8-byte alignment, not 8-byte data. That does seem to go against what the struct documentation says, "Padding is only automatically added between successive structure members. No padding is added at the beginning or the end of the encoded struct." > The 'L' in struct is documented for 3.7 to use 4 bytes, but in fact uses 8, on fedora 31. Doc bug? > >>>> x=struct.pack('L',0x102030405) >>>> x > b'\x05\x04\x03\x02\x01\x00\x00\x00' > > Given I have exact control with b, h, i, and q but L is not fixed in size I'm not sure how it can be used with certainty across OS and versions. > > Barry > Actually, you DON'T have exact control with those sizes, it just happens that all the platforms you are using happen to have the same size for those types. Welcome to the ambiguity in the C type system, the basic types are NOT fixed in size. L means 'Long' and as Christian said, that is 8 byte long on Linux-64 bit. 'L' is exactly the right type for interfacing with a routine defined as taking a long. The issue is that you don't know what type a int32_t will be (it might be int, or it might be long, and long might not be 32 bits, it will be at least 32 bits). Perhaps array could be extended so that it took '4' for a 4 byte integer and '8' for an 8 byte integer (maybe 'U4' and 'U8' for unsigned). Might as well also allow 1 and 2 for completeness for char and short (but those are currently consistent). -- Richard Damon From maisarah at avlinfotech.net Mon Dec 2 21:29:11 2019 From: maisarah at avlinfotech.net (Maisarah) Date: Tue, 03 Dec 2019 10:29:11 +0800 Subject: Python Error Still Occured on sklearn In-Reply-To: <16ec5f25a9c.f89474fd188788.8117256455982324362@avlinfotech.net> References: <16ec5f25a9c.f89474fd188788.8117256455982324362@avlinfotech.net> Message-ID: <16ec99711a1.129d57e36335551.7298109250136335338@avlinfotech.net> Thank you. Maisarah Binti Mohd Yusak Certified CPRE-FL & CTFL Software Tester, IT Team. AVL Infotech (Malaysia) Sdn. Bhd. ? L2-I-3, Enterprise - 4 , Technology Park Malaysia, Bukit Jalil, Kuala Lumpur, Malaysia -57000 Mobile: +6016 507 3051 Mail:?mailto:maisarah at avlinfotech.net LinkedIn: https://my.linkedin.com/in/maimoyu Web:?http://www.avlinfotech.net/ ---- On Mon, 02 Dec 2019 17:30:24 +0800 Maisarah wrote ---- Dear Admin, I have install and upgrade Cython as well.? I have modified and repaired and even update the library but error is still occurred: C:\Windows\system32>pip install -U scikit-learn Collecting scikit-learn ? Using cached https://files.pythonhosted.org/packages/1e/ce/9d8c88e68af0a5b5c5d78d8d2b7bcadfd45e1d6afc863ccb9aee30765b06/scikit-learn-0.21.3.tar.gz Requirement already satisfied, skipping upgrade: numpy>=1.11.0 in c:\users\user\appdata\local\programs\python\python38\lib\site-packages (from scikit-learn) (1.17.4) Requirement already satisfied, skipping upgrade: scipy>=0.17.0 in c:\users\user\appdata\local\programs\python\python38\lib\site-packages (from scikit-learn) (1.3.3) Requirement already satisfied, skipping upgrade: joblib>=0.11 in c:\users\user\appdata\local\programs\python\python38\lib\site-packages (from scikit-learn) (0.14.0) Installing collected packages: scikit-learn ??? Running setup.py install for scikit-learn ... error ??? ERROR: Command errored out with exit status 1: ???? command: 'c:\users\user\appdata\local\programs\python\python38\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\user\\AppData\\Local\\Temp\\pip-install-caioz9bv\\scikit-learn\\setup.py'"'"'; __file__='"'"'C:\\Users\\user\\AppData\\Local\\Temp\\pip-install-caioz9bv\\scikit-learn\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\user\AppData\Local\Temp\pip-record-o9y5q4bk\install-record.txt' --single-version-externally-managed --compile ???????? cwd: C:\Users\user\AppData\Local\Temp\pip-install-caioz9bv\scikit-learn\ ??? Complete output (44 lines): ??? Partial import of sklearn during the build process. ??? No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils ??? Traceback (most recent call last): ????? File "c:\users\user\appdata\local\programs\python\python38\lib\site-packages\setuptools\msvc.py", line 489, in _find_latest_available_vc_ver ??????? return self.find_available_vc_vers()[-1] ??? IndexError: list index out of range ??? During handling of the above exception, another exception occurred: ??? Traceback (most recent call last): ????? File "", line 1, in ????? File "C:\Users\user\AppData\Local\Temp\pip-install-caioz9bv\scikit-learn\setup.py", line 290, in ??????? setup_package() ????? File "C:\Users\user\AppData\Local\Temp\pip-install-caioz9bv\scikit-learn\setup.py", line 286, in setup_package ??????? setup(**metadata) ????? File "c:\users\user\appdata\local\programs\python\python38\lib\site-packages\numpy\distutils\core.py", line 137, in setup ??????? config = configuration() ????? File "C:\Users\user\AppData\Local\Temp\pip-install-caioz9bv\scikit-learn\setup.py", line 174, in configuration ??????? config.add_subpackage('sklearn') ????? File "c:\users\user\appdata\local\programs\python\python38\lib\site-packages\numpy\distutils\misc_util.py", line 1033, in add_subpackage ??????? config_list = self.get_subpackage(subpackage_name, subpackage_path, ????? File "c:\users\user\appdata\local\programs\python\python38\lib\site-packages\numpy\distutils\misc_util.py", line 999, in get_subpackage ??????? config = self._get_configuration_from_setup_py( ????? File "c:\users\user\appdata\local\programs\python\python38\lib\site-packages\numpy\distutils\misc_util.py", line 941, in _get_configuration_from_setup_py ??????? config = setup_module.configuration(*args) ????? File "sklearn\setup.py", line 76, in configuration ??????? maybe_cythonize_extensions(top_path, config) ????? File "C:\Users\user\AppData\Local\Temp\pip-install-caioz9bv\scikit-learn\sklearn\_build_utils\__init__.py", line 42, in maybe_cythonize_extensions ??????? with_openmp = check_openmp_support() ????? File "C:\Users\user\AppData\Local\Temp\pip-install-caioz9bv\scikit-learn\sklearn\_build_utils\openmp_helpers.py", line 83, in check_openmp_support ??????? ccompiler.compile(['test_openmp.c'], output_dir='objects', ????? File "c:\users\user\appdata\local\programs\python\python38\lib\distutils\_msvccompiler.py", line 360, in compile ??????? self.initialize() ????? File "c:\users\user\appdata\local\programs\python\python38\lib\distutils\_msvccompiler.py", line 253, in initialize ??????? vc_env = _get_vc_env(plat_spec) ????? File "c:\users\user\appdata\local\programs\python\python38\lib\site-packages\setuptools\msvc.py", line 185, in msvc14_get_vc_env ??????? return EnvironmentInfo(plat_spec, vc_min_ver=14.0).return_env() ????? File "c:\users\user\appdata\local\programs\python\python38\lib\site-packages\setuptools\msvc.py", line 843, in __init__ ??????? self.si = SystemInfo(self.ri, vc_ver) ????? File "c:\users\user\appdata\local\programs\python\python38\lib\site-packages\setuptools\msvc.py", line 485, in __init__ ??????? self.vc_ver = vc_ver or self._find_latest_available_vc_ver() ????? File "c:\users\user\appdata\local\programs\python\python38\lib\site-packages\setuptools\msvc.py", line 492, in _find_latest_available_vc_ver ??????? raise distutils.errors.DistutilsPlatformError(err) ??? distutils.errors.DistutilsPlatformError: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": https://visualstudio.microsoft.com/downloads/ ??? ---------------------------------------- ERROR: Command errored out with exit status 1: 'c:\users\user\appdata\local\programs\python\python38\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\user\\AppData\\Local\\Temp\\pip-install-caioz9bv\\scikit-learn\\setup.py'"'"'; __file__='"'"'C:\\Users\\user\\AppData\\Local\\Temp\\pip-install-caioz9bv\\scikit-learn\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\user\AppData\Local\Temp\pip-record-o9y5q4bk\install-record.txt' --single-version-externally-managed --compile Check the logs for full command output. C:\Windows\system32>pip install --upgrade -U scikit-learn Collecting scikit-learn ? Using cached https://files.pythonhosted.org/packages/1e/ce/9d8c88e68af0a5b5c5d78d8d2b7bcadfd45e1d6afc863ccb9aee30765b06/scikit-learn-0.21.3.tar.gz Requirement already satisfied, skipping upgrade: numpy>=1.11.0 in c:\users\user\appdata\local\programs\python\python38\lib\site-packages (from scikit-learn) (1.17.4) Requirement already satisfied, skipping upgrade: scipy>=0.17.0 in c:\users\user\appdata\local\programs\python\python38\lib\site-packages (from scikit-learn) (1.3.3) Requirement already satisfied, skipping upgrade: joblib>=0.11 in c:\users\user\appdata\local\programs\python\python38\lib\site-packages (from scikit-learn) (0.14.0) Installing collected packages: scikit-learn ??? Running setup.py install for scikit-learn ... error ??? ERROR: Command errored out with exit status 1: ???? command: 'c:\users\user\appdata\local\programs\python\python38\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\user\\AppData\\Local\\Temp\\pip-install-48dlq_5d\\scikit-learn\\setup.py'"'"'; __file__='"'"'C:\\Users\\user\\AppData\\Local\\Temp\\pip-install-48dlq_5d\\scikit-learn\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\user\AppData\Local\Temp\pip-record-arwze1t4\install-record.txt' --single-version-externally-managed --compile ???????? cwd: C:\Users\user\AppData\Local\Temp\pip-install-48dlq_5d\scikit-learn\ ??? Complete output (44 lines): ??? Partial import of sklearn during the build process. ??? No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils ??? Traceback (most recent call last): ????? File "c:\users\user\appdata\local\programs\python\python38\lib\site-packages\setuptools\msvc.py", line 489, in _find_latest_available_vc_ver ??????? return self.find_available_vc_vers()[-1] ??? IndexError: list index out of range ??? During handling of the above exception, another exception occurred: ??? Traceback (most recent call last): ????? File "", line 1, in ????? File "C:\Users\user\AppData\Local\Temp\pip-install-48dlq_5d\scikit-learn\setup.py", line 290, in ??????? setup_package() ????? File "C:\Users\user\AppData\Local\Temp\pip-install-48dlq_5d\scikit-learn\setup.py", line 286, in setup_package ??????? setup(**metadata) ????? File "c:\users\user\appdata\local\programs\python\python38\lib\site-packages\numpy\distutils\core.py", line 137, in setup ??????? config = configuration() ????? File "C:\Users\user\AppData\Local\Temp\pip-install-48dlq_5d\scikit-learn\setup.py", line 174, in configuration ??????? config.add_subpackage('sklearn') ????? File "c:\users\user\appdata\local\programs\python\python38\lib\site-packages\numpy\distutils\misc_util.py", line 1033, in add_subpackage ??????? config_list = self.get_subpackage(subpackage_name, subpackage_path, ????? File "c:\users\user\appdata\local\programs\python\python38\lib\site-packages\numpy\distutils\misc_util.py", line 999, in get_subpackage ??????? config = self._get_configuration_from_setup_py( ????? File "c:\users\user\appdata\local\programs\python\python38\lib\site-packages\numpy\distutils\misc_util.py", line 941, in _get_configuration_from_setup_py ??????? config = setup_module.configuration(*args) ????? File "sklearn\setup.py", line 76, in configuration ??????? maybe_cythonize_extensions(top_path, config) ????? File "C:\Users\user\AppData\Local\Temp\pip-install-48dlq_5d\scikit-learn\sklearn\_build_utils\__init__.py", line 42, in maybe_cythonize_extensions ??????? with_openmp = check_openmp_support() ????? File "C:\Users\user\AppData\Local\Temp\pip-install-48dlq_5d\scikit-learn\sklearn\_build_utils\openmp_helpers.py", line 83, in check_openmp_support ??????? ccompiler.compile(['test_openmp.c'], output_dir='objects', ????? File "c:\users\user\appdata\local\programs\python\python38\lib\distutils\_msvccompiler.py", line 360, in compile ??????? self.initialize() ????? File "c:\users\user\appdata\local\programs\python\python38\lib\distutils\_msvccompiler.py", line 253, in initialize ??????? vc_env = _get_vc_env(plat_spec) ????? File "c:\users\user\appdata\local\programs\python\python38\lib\site-packages\setuptools\msvc.py", line 185, in msvc14_get_vc_env ??????? return EnvironmentInfo(plat_spec, vc_min_ver=14.0).return_env() ????? File "c:\users\user\appdata\local\programs\python\python38\lib\site-packages\setuptools\msvc.py", line 843, in __init__ ??????? self.si = SystemInfo(self.ri, vc_ver) ????? File "c:\users\user\appdata\local\programs\python\python38\lib\site-packages\setuptools\msvc.py", line 485, in __init__ ??????? self.vc_ver = vc_ver or self._find_latest_available_vc_ver() ????? File "c:\users\user\appdata\local\programs\python\python38\lib\site-packages\setuptools\msvc.py", line 492, in _find_latest_available_vc_ver ??????? raise distutils.errors.DistutilsPlatformError(err) ??? distutils.errors.DistutilsPlatformError: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": https://visualstudio.microsoft.com/downloads/ ??? ---------------------------------------- ERROR: Command errored out with exit status 1: 'c:\users\user\appdata\local\programs\python\python38\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\user\\AppData\\Local\\Temp\\pip-install-48dlq_5d\\scikit-learn\\setup.py'"'"'; __file__='"'"'C:\\Users\\user\\AppData\\Local\\Temp\\pip-install-48dlq_5d\\scikit-learn\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\user\AppData\Local\Temp\pip-record-arwze1t4\install-record.txt' --single-version-externally-managed --compile Check the logs for full command output. C:\Windows\system32> Kindly advice in English language, please. Thank you. Maisarah Binti Mohd Yusak Certified CPRE-FL & CTFL Software Tester, IT Team. AVL Infotech (Malaysia) Sdn. Bhd. ? L2-I-3, Enterprise - 4 , Technology Park Malaysia, Bukit Jalil, Kuala Lumpur, Malaysia -57000 Mobile: +6016 507 3051 Mail:?mailto:maisarah at avlinfotech.net LinkedIn: https://my.linkedin.com/in/maimoyu Web:?http://www.avlinfotech.net/ From veek at dont-use-this.com Tue Dec 3 04:19:28 2019 From: veek at dont-use-this.com (Veek M) Date: Tue, 3 Dec 2019 09:19:28 -0000 (UTC) Subject: Extending property using a Subclass - single method - why Super(Baz, Baz).name.__set__ ? Message-ID: class Foo(object): @property def name(self): if hasattr(self, '_name'): print('Foo name', self._name) return self._name else: return 'default' @name.setter def name(self, value): print('Foo', self) self._name = value print(self._name) @name.deleter def name(self): print('del') self._name = None print('Foo', name) class Baz(Foo): @property def name(self): print('Baz wrapper around getter') return super().name @Foo.name.setter def name(self, value): print('Baz wrapper around setter') print(self) print(super(Baz,Baz).name, value) return super(Baz, Baz).name.__set__(self, value) b = Baz() print('print', b.name) b.name = 'v' print(b.name) Why do we user super(Baz, Baz) - are we setting a class variable called Baz.name which would trigger Baz._name = value? We are essentially doing: Foo.name.__set__(Baz, value) ? How come 'self' is not used.. like in the traditional property way where we pass an instance reference instead of a class? From __peter__ at web.de Tue Dec 3 06:16:12 2019 From: __peter__ at web.de (Peter Otten) Date: Tue, 03 Dec 2019 12:16:12 +0100 Subject: Extending property using a Subclass - single method - why Super(Baz, Baz).name.__set__ ? References: Message-ID: Veek M wrote: > class Foo(object): > @property > def name(self): > if hasattr(self, '_name'): > print('Foo name', self._name) > return self._name > else: > return 'default' > > @name.setter > def name(self, value): > print('Foo', self) > self._name = value > print(self._name) > > @name.deleter > def name(self): > print('del') > self._name = None > > print('Foo', name) > > class Baz(Foo): > @property > def name(self): > print('Baz wrapper around getter') > return super().name > > @Foo.name.setter This looks like a bug as the read-only property defined above is overwritten by a copy of the name poperty of the base class (with an updated setter). > def name(self, value): > print('Baz wrapper around setter') > print(self) > print(super(Baz,Baz).name, value) > return super(Baz, Baz).name.__set__(self, value) When you want to invoke a method of the base class property you have to look up that property in the base class. This is what super(Baz, Baz).name does. super().name for read access is an exception. The __get__() method is invoked implicitly which is probably also the reason why you cannot write super().name.somepropertymethod(...) for anything else. > > b = Baz() > print('print', b.name) > b.name = 'v' > print(b.name) > > Why do we user super(Baz, Baz) - are we setting a class variable called > Baz.name which would trigger Baz._name = value? > > We are essentially doing: > Foo.name.__set__(Baz, value) ? > > How come 'self' is not used.. like in the traditional property way where > we pass an instance reference instead of a class? From veek at dont-use-this.com Tue Dec 3 08:36:55 2019 From: veek at dont-use-this.com (Veek M) Date: Tue, 3 Dec 2019 13:36:55 -0000 (UTC) Subject: Extending property using a Subclass - single method - why Super(Baz, Baz).name.__set__ ? References: Message-ID: you've misunderstood my question, let me try again: So this is a simple descriptor class and as you can see, dunder-set needs 3 args: the descriptor CONTAINER/Bar-instance is the first arg, then a reference to the using instance/Foo-instance class Bar(object): def __set__(self, instance, value): #b-instance of Bar, f-instance of Foo, value print(self, instance, value) class Foo(object): b = Bar() f = Foo() print(f) f.b = 10 1. Now when we create/use @property.. what is the first and second argument to dunder-set (my guess is, the @property is the first arg and the second arg is 'Foo' IF you do class Foo(object): @property def whatever.. Am I right? Is there a way to check? 2. The Class Bar/descriptor acts a wrapper/protector for some sekret _var and therefore it gets all the data needed to make a judgement call.. that is, it's own name/instance-ref and the using class/instance-name-ref Note that he's receiving instance-references therefore when I start sub-classing a property why does he then switch to class-references/class-variables From aishan0403 at gmail.com Tue Dec 3 08:41:18 2019 From: aishan0403 at gmail.com (A S) Date: Tue, 3 Dec 2019 05:41:18 -0800 (PST) Subject: Extract sentences in nested parentheses using Python In-Reply-To: References: <7a365fa0-e721-4a95-8c30-d3661301a7b2@googlegroups.com> Message-ID: On Tuesday, 3 December 2019 01:01:25 UTC+8, Peter Otten wrote: > A S wrote: > > I think I've seen this question before ;) > > > I am trying to extract all strings in nested parentheses (along with the > > parentheses itself) in my .txt file. Please see the sample .txt file that > > I have used in this example here: > > (https://drive.google.com/open?id=1UKc0ZgY9Fsz5O1rSeBCLqt5dwZkMaQgr). > > > > I have tried and done up three different codes but none of them seems to > > be able to extract all the nested parentheses. They can only extract a > > portion of the nested parentheses. Any advice on what I've done wrong > > could really help! > > > > Here are the three codes I have done so far: > > > > 1st attempt: > > > > import re > > from os.path import join > > > > def balanced_braces(args): > > parts = [] > > for arg in args: > > if '(' not in arg: > > continue > > There could still be a ")" that you miss > > > chars = [] > > n = 0 > > for c in arg: > > if c == '(': > > if n > 0: > > chars.append(c) > > n += 1 > > elif c == ')': > > n -= 1 > > if n > 0: > > chars.append(c) > > elif n == 0: > > parts.append(''.join(chars).lstrip().rstrip()) > > chars = [] > > elif n > 0: > > chars.append(c) > > return parts > > It's probably easier to understand and implement when you process the > complete text at once. Then arbitrary splits don't get in the way of your > quest for ( and ). You just have to remember the position of the first > opening ( and number of opening parens that have to be closed before you > take the complete expression: > > level: 00011112222100 > text: abc(def(gh))ij > when we are here^ > we need^ > > A tentative implementation: > > $ cat parse.py > import re > > NOT_SET = object() > > def scan(text): > level = 0 > start = NOT_SET > for m in re.compile("[()]").finditer(text): > if m.group() == ")": > level -= 1 > if level < 0: > raise ValueError("underflow: more closing than opening > parens") > if level == 0: > # outermost closing parenthesis: > # deliver enclosed string including parens. > yield text[start:m.end()] > start = NOT_SET > elif m.group() == "(": > if level == 0: > # outermost opening parenthesis: remember position. > assert start is NOT_SET > start = m.start() > level += 1 > else: > assert False > if level > 0: > raise ValueError("unclosed parens remain") > > > if __name__ == "__main__": > with open("lan sample text file.txt") as instream: > text = instream.read() > for chunk in scan(text): > print(chunk) > $ python3 parse.py > ("xE'", PUT(xx.xxxx.),"'") > ("TRUuuuth") Hello Peter! I tried this on my actual working files and it returned this error: "unclosed parens remain". In this case, how can I continue to parse through my text files by only extracting those with balanced parentheses and ignore those that are incomplete? From __peter__ at web.de Tue Dec 3 10:47:49 2019 From: __peter__ at web.de (Peter Otten) Date: Tue, 03 Dec 2019 16:47:49 +0100 Subject: Extract sentences in nested parentheses using Python References: <7a365fa0-e721-4a95-8c30-d3661301a7b2@googlegroups.com> Message-ID: A S wrote: > On Tuesday, 3 December 2019 01:01:25 UTC+8, Peter Otten wrote: >> A S wrote: >> >> I think I've seen this question before ;) >> >> > I am trying to extract all strings in nested parentheses (along with >> > the parentheses itself) in my .txt file. Please see the sample .txt >> > file that I have used in this example here: >> > (https://drive.google.com/open?id=1UKc0ZgY9Fsz5O1rSeBCLqt5dwZkMaQgr). >> > >> > I have tried and done up three different codes but none of them seems >> > to be able to extract all the nested parentheses. They can only extract >> > a portion of the nested parentheses. Any advice on what I've done wrong >> > could really help! >> > >> > Here are the three codes I have done so far: >> > >> > 1st attempt: >> > >> > import re >> > from os.path import join >> > >> > def balanced_braces(args): >> > parts = [] >> > for arg in args: >> > if '(' not in arg: >> > continue >> >> There could still be a ")" that you miss >> >> > chars = [] >> > n = 0 >> > for c in arg: >> > if c == '(': >> > if n > 0: >> > chars.append(c) >> > n += 1 >> > elif c == ')': >> > n -= 1 >> > if n > 0: >> > chars.append(c) >> > elif n == 0: >> > parts.append(''.join(chars).lstrip().rstrip()) >> > chars = [] >> > elif n > 0: >> > chars.append(c) >> > return parts >> >> It's probably easier to understand and implement when you process the >> complete text at once. Then arbitrary splits don't get in the way of your >> quest for ( and ). You just have to remember the position of the first >> opening ( and number of opening parens that have to be closed before you >> take the complete expression: >> >> level: 00011112222100 >> text: abc(def(gh))ij >> when we are here^ >> we need^ >> >> A tentative implementation: >> >> $ cat parse.py >> import re >> >> NOT_SET = object() >> >> def scan(text): >> level = 0 >> start = NOT_SET >> for m in re.compile("[()]").finditer(text): >> if m.group() == ")": >> level -= 1 >> if level < 0: >> raise ValueError("underflow: more closing than opening >> parens") >> if level == 0: >> # outermost closing parenthesis: >> # deliver enclosed string including parens. >> yield text[start:m.end()] >> start = NOT_SET >> elif m.group() == "(": >> if level == 0: >> # outermost opening parenthesis: remember position. >> assert start is NOT_SET >> start = m.start() >> level += 1 >> else: >> assert False >> if level > 0: >> raise ValueError("unclosed parens remain") >> >> >> if __name__ == "__main__": >> with open("lan sample text file.txt") as instream: >> text = instream.read() >> for chunk in scan(text): >> print(chunk) >> $ python3 parse.py >> ("xE'", PUT(xx.xxxx.),"'") >> ("TRUuuuth") > > Hello Peter! I tried this on my actual working files and it returned this > error: "unclosed parens remain". In this case, how can I continue to parse > through my text files by only extracting those with balanced parentheses > and ignore those that are incomplete? filenames = ... for filename in filenames: with open(filename) as instream: text = instream.read() try: chunks = list(scan(text)) except ValueError as err: print(f"{err} in file {filename!r}", file=sys.stderr) else: for chunk in chunks: print(chunk) From geremy85 at gmail.com Mon Dec 2 23:58:11 2019 From: geremy85 at gmail.com (geremy85 at gmail.com) Date: Mon, 2 Dec 2019 20:58:11 -0800 (PST) Subject: lxml question -- creating an etree.Element attribute with ':' in the name In-Reply-To: References: Message-ID: <0fc93031-c19f-47ea-9539-c4d94e12314a@googlegroups.com> Theanks a lot From Karsten.Hilbert at gmx.net Tue Dec 3 11:00:45 2019 From: Karsten.Hilbert at gmx.net (Karsten Hilbert) Date: Tue, 3 Dec 2019 17:00:45 +0100 Subject: lxml question -- creating an etree.Element attribute with ':' in the name In-Reply-To: <0fc93031-c19f-47ea-9539-c4d94e12314a@googlegroups.com> References: <0fc93031-c19f-47ea-9539-c4d94e12314a@googlegroups.com> Message-ID: <20191203160044.GB1277@hermes.hilbert.loc> On Mon, Dec 02, 2019 at 08:58:11PM -0800, geremy85 at gmail.com wrote: > Date: Mon, 2 Dec 2019 20:58:11 -0800 (PST) > From: geremy85 at gmail.com > To: python-list at python.org > Subject: Re: lxml question -- creating an etree.Element attribute with ':' > in the name > User-Agent: G2/1.0 > > Theanks a lot > -- > https://mail.python.org/mailman/listinfo/python-list you are welcoem -- GPG 40BE 5B0E C98E 1713 AFA6 5BC0 3BEA AC80 7D4F C89B From __peter__ at web.de Tue Dec 3 11:24:17 2019 From: __peter__ at web.de (Peter Otten) Date: Tue, 03 Dec 2019 17:24:17 +0100 Subject: Extending property using a Subclass - single method - why Super(Baz, Baz).name.__set__ ? References: Message-ID: Veek M wrote: > you've misunderstood my question There were a lot of foobars bazzing in my head, but at least I tried ;) > , let me try again: > > So this is a simple descriptor class and as you can see, dunder-set needs > 3 args: the descriptor CONTAINER/Bar-instance is the first arg, then a > reference to the using instance/Foo-instance > > class Bar(object): > > def __set__(self, instance, value): > #b-instance of Bar, f-instance of Foo, value > print(self, instance, value) > > > class Foo(object): > b = Bar() > > f = Foo() > print(f) > f.b = 10 > > 1. Now when we create/use @property.. > what is the first and second argument to dunder-set > > (my guess is, the @property is the first arg and the second arg is 'Foo' > IF you do > > class Foo(object): > @property > def whatever.. > > Am I right? Is there a way to check? @foo def bar(...): ... is an alternative way to writing def bar(...): ... bar = foo(bar) so you can just add conforming __init__() and __get__() methods to your Bar descriptor and see for yourself: >> class Bar: ... def __init__(self, fget): pass ... def __set__(*args): print(args) ... def __get__(*args): pass ... >>> class Foo: ... @Bar ... def whatever(self): pass ... >>> foo = Foo() >>> foo.whatever = 42 (<__main__.Bar object at 0x7f92b55d70b8>, <__main__.Foo object at 0x7f92b55d70f0>, 42) > 2. The Class Bar/descriptor acts a wrapper/protector for some sekret _var > and therefore it gets all the data needed to make a judgement call.. that > is, it's own name/instance-ref and the using class/instance-name-ref > > Note that he's receiving instance-references > > therefore when I start sub-classing a property why does he then switch to > class-references/class-variables If you were subclassing a property you'd do class my_property(property): # tinker with __get/set/whatnot__ When you want to wrap a property defined in a superclass the property instance is not part of the class hierarchy that you are interested in. Instead of calling super() it has to figure out the base class of its "host" class manually. If you want overridable properties you can devise a way to look up the method: >>> class Foo: ... @property ... def name(self): return self.get_name() ... @name.setter ... def name(self, value): self.set_name(value) ... def set_name(self, value): self._name = value ... def get_name(self): return self._name ... >>> class Bar(Foo): ... def set_name(self, value): ... print("setting") ... super().set_name(value) ... def get_name(self): ... print("getting") ... return super().get_name() ... >>> foo = Foo() >>> foo.name = "bar" >>> foo.name 'bar' >>> bar = Bar() >>> bar.name = "baz" setting >>> bar.name getting 'baz' From python.list at tim.thechases.com Tue Dec 3 11:23:08 2019 From: python.list at tim.thechases.com (Tim Chase) Date: Tue, 3 Dec 2019 10:23:08 -0600 Subject: increasing the page size of a dbm store? In-Reply-To: <4dd1ac29-8803-a1bd-3158-0cd90f392c48@gmail.com> References: <20191126192432.535ee241@bigbox.attlocal.net> <0EAFC9E0-02D2-4745-99B7-654B7A445EDD@barrys-emacs.org> <20191201205016.577673f9@bigbox.attlocal.net> <4dd1ac29-8803-a1bd-3158-0cd90f392c48@gmail.com> Message-ID: <20191203102308.1c0911c8@bigbox.attlocal.net> On 2019-12-02 16:49, Michael Torrie wrote: > On 12/1/19 7:50 PM, Tim Chase wrote: > > After sparring with it a while, I tweaked the existing job so > > that it chunked things into dbm-appropriate sizes to limp > > through; for the subsequent job (where I would have used dbm > > again) I went ahead and switched to sqlite and had no further > > issues. > > How did you replace a key/value store with a relational database? > Is a SQLite database fast enough at this sort of thing that it > wasn't really designed for? It was certainly slower, though it wasn't so bad once I had proper indexing and submitted queries that pulled back multiple results in one query. But even with the slightly slower run-time aspects, it was still faster than starting a job (expecting it to run to completion overnight), having it crash, manually deleting my cache, and manually resuming from where it left off, all multiple times. And all said, since it was network I/O bound, once I had the populated cache (resulting cache.db file was about 1TB...thank goodness for transparent compression with ZFS), turnarounds took more like 30min rather than 3 days. More the "go work on something else and come back" than the "let it run overnight". -tkc From rgaddi at highlandtechnology.invalid Tue Dec 3 13:01:22 2019 From: rgaddi at highlandtechnology.invalid (Rob Gaddi) Date: Tue, 3 Dec 2019 10:01:22 -0800 Subject: array and struct 64-bit Linux change in behavior Python 3.7 and 2.7 In-Reply-To: References: <475624D9-15D8-4A82-A8B4-5A174BA87DCD@barrys-emacs.org> <2b3d0c1a-aab4-9084-1011-975f911352d8@Damon-Family.org> Message-ID: On 12/2/19 5:50 PM, Richard Damon wrote: > > Perhaps array could be extended so that it took '4' for a 4 byte integer > and '8' for an 8 byte integer (maybe 'U4' and 'U8' for unsigned). Might > as well also allow 1 and 2 for completeness for char and short (but > those are currently consistent). > I will note that numpy arrays give exactly this level of control, as do ctypes arrays. The standard array library might just be the wrong tool for the job of reading binary data. From Chris.Clark at actian.com Wed Dec 4 00:15:05 2019 From: Chris.Clark at actian.com (Chris Clark) Date: Wed, 4 Dec 2019 05:15:05 +0000 Subject: array and struct 64-bit Linux change in behavior Python 3.7 and 2.7 In-Reply-To: <2b3d0c1a-aab4-9084-1011-975f911352d8@Damon-Family.org> References: <475624D9-15D8-4A82-A8B4-5A174BA87DCD@barrys-emacs.org> <2b3d0c1a-aab4-9084-1011-975f911352d8@Damon-Family.org> Message-ID: Thanks for all the replies (and apologies for top posting, I have a brain dead email client ?). I think the consensus from the various threads is that the docs are either lacking or misleading. I mentioned that this impacts bytes and the problem there is more telling as it hard fails (this is how I first discovered this was an issue): >>> array.array('L', b'\0\0\0\0') Traceback (most recent call last): File "", line 1, in ValueError: string length not a multiple of item size I don't believe the documentation is accurate by using the word "minimum". Minimum would suggest that it would accept a 4-byte value as a minimum and on 64-bit it does *not*, it hard fails. If it were to document that, "the sizes are native integer types for the platform, the table documents some typical but *not* guaranteed sizes", that would be more clear. For struct - I think the '<' and '=' non-padding docs could benefit from some explanation.. I'm not sure what yet ? I saw a few suggestions on alternatives for size specifications, I'm definitely in favor of that (right now I'm probing I and L to determine size before using them for real). I don?t think U prefix would work as array really only accepts a single specifier. If array was to be updated to use multiple character specifiers I would recommend matching the struct specifier (which it is close to at the moment) format. For my uses case I'm seriously thinking about not using array moving forward and only using struct. I briefly wondered about ctypes (it has nice names, e.g. c_int64 that are unambiguous) but then I remembered it is not available in Jython). With the benefit of hindsight it would have been better if array (and struct) used stdint.h types, those types and lengths are explicitly documented. Regarding Barry's comment, yep size consistency with array is a pain - what I implemented as workaround is below (and likely to be my solution going forward): x = array.array('L', [0]) if x.itemsize == 4: FMT_ARRAY_4BYTE = 'L' FMT_STRUCT_4BYTE = ' Sent: Monday, December 2, 2019 5:50 PM To: python-list at python.org Subject: Re: array and struct 64-bit Linux change in behavior Python 3.7 and 2.7 On 12/2/19 4:25 PM, Barry Scott wrote: > >> On 2 Dec 2019, at 17:55, Rob Gaddi wrote: >> >> On 12/2/19 9:26 AM, Chris Clark wrote: >>> Test case: >>> import array >>> array.array('L', [0]) # x.itemsize == 8 rather than >>> 4 This works fine (returns 4) under Windows Python 3.7.3 64-bit >>> build. >>> Under Ubuntu; Python 2.7.15rc1, 3.6.5, 3.70b3 64-bit this returns 8. Documentation at https://docs.python.org/3/library/array.html explicitly states 'L' is for size 4. >>> It impacts all uses types of array (e.g. reading from byte strings). >>> The struct module is a little different: >>> import struct >>> x = struct.pack('L', 0) >>> # len(x) ===8 rather than 4 >>> This can be worked around by using '=L' - which is not well documented - so this maybe a doc issue. >>> Wanted to post here for comments before opening a bug at >>> https://bugs.python.org/ >>> Is anyone seeing this under Debian/Ubuntu? >>> Chris >> I'd say not a bug, at least in array. Reading that array documentation you linked, 4 is explicitly the MINIMUM size in bytes, not the guaranteed size. > I'm wondering how useful it is that for array you can read from a file but have no ideas how many bytes each item needs. > If I have a file with int32_t in it I cannot from the docs know how to read that file into an array. > >> The struct situation is, as you said, a bit different. I believe that with the default native alignment @, you're seeing 4-byte data padded to an 8-byte alignment, not 8-byte data. That does seem to go against what the struct documentation says, "Padding is only automatically added between successive structure members. No padding is added at the beginning or the end of the encoded struct." > The 'L' in struct is documented for 3.7 to use 4 bytes, but in fact uses 8, on fedora 31. Doc bug? > >>>> x=struct.pack('L',0x102030405) >>>> x > b'\x05\x04\x03\x02\x01\x00\x00\x00' > > Given I have exact control with b, h, i, and q but L is not fixed in size I'm not sure how it can be used with certainty across OS and versions. > > Barry > Actually, you DON'T have exact control with those sizes, it just happens that all the platforms you are using happen to have the same size for those types. Welcome to the ambiguity in the C type system, the basic types are NOT fixed in size. L means 'Long' and as Christian said, that is 8 byte long on Linux-64 bit. 'L' is exactly the right type for interfacing with a routine defined as taking a long. The issue is that you don't know what type a int32_t will be (it might be int, or it might be long, and long might not be 32 bits, it will be at least 32 bits). Perhaps array could be extended so that it took '4' for a 4 byte integer and '8' for an 8 byte integer (maybe 'U4' and 'U8' for unsigned). Might as well also allow 1 and 2 for completeness for char and short (but those are currently consistent). -- Richard Damon From rosuav at gmail.com Wed Dec 4 00:48:42 2019 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 4 Dec 2019 16:48:42 +1100 Subject: array and struct 64-bit Linux change in behavior Python 3.7 and 2.7 In-Reply-To: References: <475624D9-15D8-4A82-A8B4-5A174BA87DCD@barrys-emacs.org> <2b3d0c1a-aab4-9084-1011-975f911352d8@Damon-Family.org> Message-ID: On Wed, Dec 4, 2019 at 4:16 PM Chris Clark wrote: > I think the consensus from the various threads is that the docs are either lacking or misleading. > > I mentioned that this impacts bytes and the problem there is more telling as it hard fails (this is how I first discovered this was an issue): > > >>> array.array('L', b'\0\0\0\0') > Traceback (most recent call last): > File "", line 1, in > ValueError: string length not a multiple of item size > > I don't believe the documentation is accurate by using the word "minimum". Minimum would suggest that it would accept a 4-byte value as a minimum and on 64-bit it does *not*, it hard fails. If it were to document that, "the sizes are native integer types for the platform, the table documents some typical but *not* guaranteed sizes", that would be more clear. > I think array.array() is possibly the wrong tool for this job. If you have a collection of bytes from some well-defined source (eg you're parsing a file in a known format), struct is better suited to it, because it's easy to define both the size and byte order. > For my uses case I'm seriously thinking about not using array moving forward and only using struct. I briefly wondered about ctypes (it has nice names, e.g. c_int64 that are unambiguous) but then I remembered it is not available in Jython). > I wouldn't bother with ctypes for this type of job. > Regarding Barry's comment, yep size consistency with array is a pain - what I implemented as workaround is below (and likely to be my solution going forward): > > x = array.array('L', [0]) > if x.itemsize == 4: > FMT_ARRAY_4BYTE = 'L' > FMT_STRUCT_4BYTE = ' else: > x = array.array('I', [0]) > if x.itemsize == 4: > FMT_ARRAY_4BYTE = 'I' > FMT_STRUCT_4BYTE = ' del(x) > > and then use the constants in array/struct calls where (binary) file IO is happening. Yep, looks like struct is the way to go here. (Especially since you don't have a final 'else'.) ChrisA From p4j at j4d.net Wed Dec 4 08:21:46 2019 From: p4j at j4d.net (Pankaj Jangid) Date: Wed, 04 Dec 2019 18:51:46 +0530 Subject: Developers are advised to purge these malicious packages Message-ID: ``` The Python security team removed two trojanized Python libraries from PyPI (Python Package Index) that were caught stealing SSH and GPG keys from the projects of infected developers. The first is "python3-dateutil," which imitated the popular "dateutil" library. The second is "jeIlyfish" (the first L is an I), which mimicked the "jellyfish" library. ``` https://www.zdnet.com/article/two-malicious-python-libraries-removed-from-pypi/ Regards, -- Pankaj Jangid From ast at invalid Wed Dec 4 10:18:04 2019 From: ast at invalid (ast) Date: Wed, 4 Dec 2019 16:18:04 +0100 Subject: threading Message-ID: <5de7ce2e$0$3871$426a74cc@news.free.fr> Hi An operation like x+=1 on a global variable x is not thread safe because there can be a thread switch between reading and writing to x. The correct way is to use a lock lock = threading.Lock with lock: x+=1 I tried to write a program without the lock which should fail. Here it is: import threading x = 0 def test(): global x for i in range(100): x+=1 threadings = [] for i in range(100): t = threading.Thread(target=test) threadings.append(t) t.start() for t in threadings: t.join() print(x) 10000 The result is always correct: 10000 Why ? Secondly, how the switch between threads is done by the processor ? Is there a hardware interrupt coming from a timer ? From David.Raymond at tomtom.com Wed Dec 4 10:47:57 2019 From: David.Raymond at tomtom.com (David Raymond) Date: Wed, 4 Dec 2019 15:47:57 +0000 Subject: threading In-Reply-To: <5de7ce2e$0$3871$426a74cc@news.free.fr> References: <5de7ce2e$0$3871$426a74cc@news.free.fr> Message-ID: 100 increments happen very fast, and means each thread will probably complete before the main thread has even started the next one. Bump that up to 1_000_000 or so and you'll probably trigger it. I did a test with a print(x) at the start of test() to see what the number was when each thread kicked off, and the very first thread had got it up to 655,562 by the time the second thread had started and gotten to that print statement. -----Original Message----- From: Python-list On Behalf Of ast Sent: Wednesday, December 4, 2019 10:18 AM To: python-list at python.org Subject: threading Hi An operation like x+=1 on a global variable x is not thread safe because there can be a thread switch between reading and writing to x. The correct way is to use a lock lock = threading.Lock with lock: x+=1 I tried to write a program without the lock which should fail. Here it is: import threading x = 0 def test(): global x for i in range(100): x+=1 threadings = [] for i in range(100): t = threading.Thread(target=test) threadings.append(t) t.start() for t in threadings: t.join() print(x) 10000 The result is always correct: 10000 Why ? Secondly, how the switch between threads is done by the processor ? Is there a hardware interrupt coming from a timer ? -- https://mail.python.org/mailman/listinfo/python-list From david at lowryduda.com Wed Dec 4 12:59:57 2019 From: david at lowryduda.com (David Lowry-Duda) Date: Wed, 4 Dec 2019 12:59:57 -0500 Subject: Developers are advised to purge these malicious packages In-Reply-To: References: Message-ID: <20191204175957.GA24123@mail.lowryduda.com> I notice that "python3-dateutil" is in over 4000 github repositories [1]. That sounds like a disaster. [1]: https://github.com/search?q=python3-dateutil&type=Code - DLD -- David Lowry-Duda From christian at python.org Wed Dec 4 13:17:58 2019 From: christian at python.org (Christian Heimes) Date: Wed, 4 Dec 2019 19:17:58 +0100 Subject: Developers are advised to purge these malicious packages In-Reply-To: <20191204175957.GA24123@mail.lowryduda.com> References: <20191204175957.GA24123@mail.lowryduda.com> Message-ID: On 04/12/2019 18.59, David Lowry-Duda wrote: > I notice that "python3-dateutil" is in over 4000 github repositories > [1]. That sounds like a disaster. > > [1]: https://github.com/search?q=python3-dateutil&type=Code At least the first pages are packaging files for Debian, Fedora, and other Linux distributions. Downstream distributions provide a Python package under multiple names. For example the Fedora's build spec [1] creates python2-dateutil and python3-dateutil packages from the python-dateutil upstream project. Attackers abuse the fact and try to typo-squat packages in hope that somebody uses the Linux distribution package name "python3-dateutil" instead of the upstream name "python-dateutil" in requirements.txt Christian [1] https://src.fedoraproject.org/rpms/python-dateutil/blob/master/f/python-dateutil.spec From rob at despammer.com Wed Dec 4 15:25:33 2019 From: rob at despammer.com (RobH) Date: Wed, 4 Dec 2019 20:25:33 +0000 Subject: ImportError: No module named Adafruit_SSD1306 Message-ID: I am trying to do this project on a pi zero: http://frederickvandenbosch.be/?p=1365 I copied the code to the pi zero Download folder and when I run it I get the above error at line 4 Import Adafruit_SSD1306 I am using python version 2.7.16, if that makes any difference I have the same module as the authors' link goes to : Monochrome 1.3" 128x64 OLED graphic display - STEMMA QT / Qwiic Have I missed something. From torriem at gmail.com Wed Dec 4 17:34:50 2019 From: torriem at gmail.com (Michael Torrie) Date: Wed, 4 Dec 2019 15:34:50 -0700 Subject: Developers are advised to purge these malicious packages In-Reply-To: <20191204175957.GA24123@mail.lowryduda.com> References: <20191204175957.GA24123@mail.lowryduda.com> Message-ID: <32dc24a3-1c25-4497-7bdb-428c7a516f2b@gmail.com> On 12/4/19 10:59 AM, David Lowry-Duda wrote: > I notice that "python3-dateutil" is in over 4000 github repositories > [1]. That sounds like a disaster. > > [1]: https://github.com/search?q=python3-dateutil&type=Code It's clearly not, as Christian has already said. In fact it would be very difficult to determine from a github search whether this bad package was actually deployed anywhere. Since it presents a fake "dateutil" module, imports would look the same and proper as using the correct one. The only way this package comes into play is if someone pip installed it, or had an install script that installed it, or if it were bundled in the source tree. So this is very bad indeed, but not as bad as you suggest. We're not nearly as much at risk as node.js npm users are yet. From best_lay at yahoo.com Wed Dec 4 17:33:45 2019 From: best_lay at yahoo.com (Wildman) Date: Wed, 04 Dec 2019 16:33:45 -0600 Subject: ImportError: No module named Adafruit_SSD1306 References: Message-ID: On Wed, 04 Dec 2019 20:25:33 +0000, RobH wrote: > I am trying to do this project on a pi zero: > > http://frederickvandenbosch.be/?p=1365 > > I copied the code to the pi zero Download folder and when I run it I get > the above error at line 4 > Import Adafruit_SSD1306 > > I am using python version 2.7.16, if that makes any difference > I have the same module as the authors' link goes to : > > Monochrome 1.3" 128x64 OLED graphic display - STEMMA QT / Qwiic > > Have I missed something. The error indicates that Adafruit_SSD1306 in not installed. https://github.com/adafruit/Adafruit_Python_SSD1306 -- GNU/Linux user #557453 The cow died so I don't need your bull! From rob at despammer.com Wed Dec 4 18:06:44 2019 From: rob at despammer.com (RobH) Date: Wed, 4 Dec 2019 23:06:44 +0000 Subject: ImportError: No module named Adafruit_SSD1306 In-Reply-To: References: Message-ID: On 04/12/2019 22:33, Wildman wrote: > On Wed, 04 Dec 2019 20:25:33 +0000, RobH wrote: > >> I am trying to do this project on a pi zero: >> >> http://frederickvandenbosch.be/?p=1365 >> >> I copied the code to the pi zero Download folder and when I run it I get >> the above error at line 4 >> Import Adafruit_SSD1306 >> >> I am using python version 2.7.16, if that makes any difference >> I have the same module as the authors' link goes to : >> >> Monochrome 1.3" 128x64 OLED graphic display - STEMMA QT / Qwiic >> >> Have I missed something. > > The error indicates that Adafruit_SSD1306 in not installed. > > https://github.com/adafruit/Adafruit_Python_SSD1306 > I have the library in the same Downloads folder, but I don't know how to actually install it as it doesn't have an .sh file included From python at python.invalid Wed Dec 4 18:15:28 2019 From: python at python.invalid (Python) Date: Thu, 5 Dec 2019 00:15:28 +0100 Subject: ImportError: No module named Adafruit_SSD1306 In-Reply-To: References: Message-ID: <5de83dea$0$31397$426a74cc@news.free.fr> Le 05/12/2019 ? 00:06, RobH a ?crit?: > On 04/12/2019 22:33, Wildman wrote: >> On Wed, 04 Dec 2019 20:25:33 +0000, RobH wrote: >> >>> I am trying to do this project on a pi zero: >>> >>> http://frederickvandenbosch.be/?p=1365 >>> >>> I copied the code to the pi zero Download folder and when I run it I get >>> the above error at line 4 >>> Import Adafruit_SSD1306 >>> >>> I am using python version 2.7.16, if that makes any difference >>> I have the same module as the authors' link goes to : >>> >>> Monochrome 1.3" 128x64 OLED graphic display - STEMMA QT / Qwiic >>> >>> Have I missed something. >> >> The error indicates that Adafruit_SSD1306 in not installed. >> >> https://github.com/adafruit/Adafruit_Python_SSD1306 >> > > I have the library in the same Downloads folder, but I don't know how to > actually install it as it doesn't have an .sh file included What cannot you understand in the Installing section of README.md? sudo python -m pip install --upgrade pip setuptools wheel sudo pip install Adafruit-SSD1306 Or alternatively: sudo python -m pip install --upgrade pip setuptools wheel git clone https://github.com/adafruit/Adafruit_Python_SSD1306.git cd Adafruit_Python_SSD1306 sudo python setup.py install even WORSE, what cannot you undertand at the top of same file? This library has been deprecated! We are leaving this up for historical and research purposes but archiving the repository. From pahome.chen at mirlab.org Wed Dec 4 22:28:51 2019 From: pahome.chen at mirlab.org (lampahome) Date: Thu, 5 Dec 2019 11:28:51 +0800 Subject: What does the blue color section in background mean? Message-ID: I tried to plot graph about a time-series with library statsmodel. I decide to plot autocorrelation function, but I don't know the blue section in the example graph mean... Can anyone tell me? The plot_acf example link: https://www.statsmodels.org/dev/generated/statsmodels.graphics.tsaplots.plot_acf.html#statsmodels.graphics.tsaplots.plot_acf The graph in example: https://www.statsmodels.org/dev/plots/graphics_tsa_plot_acf.png thx From p4j at j4d.net Thu Dec 5 04:32:47 2019 From: p4j at j4d.net (Pankaj Jangid) Date: Thu, 05 Dec 2019 15:02:47 +0530 Subject: Developers are advised to purge these malicious packages References: <20191204175957.GA24123@mail.lowryduda.com> Message-ID: Christian Heimes writes: > On 04/12/2019 18.59, David Lowry-Duda wrote: >> I notice that "python3-dateutil" is in over 4000 github repositories >> [1]. That sounds like a disaster. >> >> [1]: https://github.com/search?q=python3-dateutil&type=Code > > At least the first pages are packaging files for Debian, Fedora, and > other Linux distributions. Downstream distributions provide a Python > package under multiple names. For example the Fedora's build spec [1] > creates python2-dateutil and python3-dateutil packages from the > python-dateutil upstream project. > > Attackers abuse the fact and try to typo-squat packages in hope that > somebody uses the Linux distribution package name "python3-dateutil" > instead of the upstream name "python-dateutil" in requirements.txt > Nice explanation. Thanks. From barry at barrys-emacs.org Thu Dec 5 04:27:43 2019 From: barry at barrys-emacs.org (Barry Scott) Date: Thu, 5 Dec 2019 09:27:43 +0000 Subject: array and struct 64-bit Linux change in behavior Python 3.7 and 2.7 In-Reply-To: <2b3d0c1a-aab4-9084-1011-975f911352d8@Damon-Family.org> References: <475624D9-15D8-4A82-A8B4-5A174BA87DCD@barrys-emacs.org> <2b3d0c1a-aab4-9084-1011-975f911352d8@Damon-Family.org> Message-ID: <8C71B143-C11A-4A1F-901E-6CEFF6D73814@barrys-emacs.org> > On 3 Dec 2019, at 01:50, Richard Damon wrote: > > On 12/2/19 4:25 PM, Barry Scott wrote: >> >>> On 2 Dec 2019, at 17:55, Rob Gaddi wrote: >>> >>> On 12/2/19 9:26 AM, Chris Clark wrote: >>>> Test case: >>>> import array >>>> array.array('L', [0]) >>>> # x.itemsize == 8 rather than 4 >>>> This works fine (returns 4) under Windows Python 3.7.3 64-bit build. >>>> Under Ubuntu; Python 2.7.15rc1, 3.6.5, 3.70b3 64-bit this returns 8. Documentation at https://docs.python.org/3/library/array.html explicitly states 'L' is for size 4. >>>> It impacts all uses types of array (e.g. reading from byte strings). >>>> The struct module is a little different: >>>> import struct >>>> x = struct.pack('L', 0) >>>> # len(x) ===8 rather than 4 >>>> This can be worked around by using '=L' - which is not well documented - so this maybe a doc issue. >>>> Wanted to post here for comments before opening a bug at https://bugs.python.org/ >>>> Is anyone seeing this under Debian/Ubuntu? >>>> Chris >>> I'd say not a bug, at least in array. Reading that array documentation you linked, 4 is explicitly the MINIMUM size in bytes, not the guaranteed size. >> I'm wondering how useful it is that for array you can read from a file but have no ideas how many bytes each item needs. >> If I have a file with int32_t in it I cannot from the docs know how to read that file into an array. >> >>> The struct situation is, as you said, a bit different. I believe that with the default native alignment @, you're seeing 4-byte data padded to an 8-byte alignment, not 8-byte data. That does seem to go against what the struct documentation says, "Padding is only automatically added between successive structure members. No padding is added at the beginning or the end of the encoded struct." >> The 'L' in struct is documented for 3.7 to use 4 bytes, but in fact uses 8, on fedora 31. Doc bug? >> >>>>> x=struct.pack('L',0x102030405) >>>>> x >> b'\x05\x04\x03\x02\x01\x00\x00\x00' >> >> Given I have exact control with b, h, i, and q but L is not fixed in size I'm not sure how it can be used with certainty across OS and versions. >> >> Barry >> > Actually, you DON'T have exact control with those sizes, it just happens > that all the platforms you are using happen to have the same size for > those types. According to the docs for struct (python 2.7 and python 3.8) I do have exact control for the types I listed. Or did I miss a caveat on that page? The docs for array indeed show that you have no exact control and that is what I'm commenting on. As other have observed that makes array the wrong tool to read data of a fixed format. > Welcome to the ambiguity in the C type system, the basic > types are NOT fixed in size. Of course that is why int32_t etc where added to the C standards. > L means 'Long' and as Christian said, that > is 8 byte long on Linux-64 bit. 'L' is exactly the right type for > interfacing with a routine defined as taking a long. The issue is that > you don't know what type a int32_t will be (it might be int, or it might > be long, and long might not be 32 bits, it will be at least 32 bits). > > Perhaps array could be extended so that it took '4' for a 4 byte integer > and '8' for an 8 byte integer (maybe 'U4' and 'U8' for unsigned). Might > as well also allow 1 and 2 for completeness for char and short (but > those are currently consistent). Personally I have never thought to use array. I have user struct and ctypes extensively and they give me the documented control I need to work with data structures and APIs. Barry > > -- > Richard Damon > > -- > https://mail.python.org/mailman/listinfo/python-list > From rob at despammer.com Thu Dec 5 05:07:28 2019 From: rob at despammer.com (RobH) Date: Thu, 5 Dec 2019 10:07:28 +0000 Subject: ImportError: No module named Adafruit_SSD1306 In-Reply-To: <5de83dea$0$31397$426a74cc@news.free.fr> References: <5de83dea$0$31397$426a74cc@news.free.fr> Message-ID: On 04/12/2019 23:15, Python wrote: > Le 05/12/2019 ? 00:06, RobH a ?crit?: >> On 04/12/2019 22:33, Wildman wrote: >>> On Wed, 04 Dec 2019 20:25:33 +0000, RobH wrote: >>> >>>> I am trying to do this project on a pi zero: >>>> >>>> http://frederickvandenbosch.be/?p=1365 >>>> >>>> I copied the code to the pi zero Download folder and when I run it I >>>> get >>>> the above error at line 4 >>>> Import Adafruit_SSD1306 >>>> >>>> I am using python version 2.7.16, if that makes any difference >>>> I have the same module as the authors' link goes to : >>>> >>>> Monochrome 1.3" 128x64 OLED graphic display - STEMMA QT / Qwiic >>>> >>>> Have I missed something. >>> >>> The error indicates that Adafruit_SSD1306 in not installed. >>> >>> https://github.com/adafruit/Adafruit_Python_SSD1306 >>> >> >> I have the library in the same Downloads folder, but I don't know how >> to actually install it as it doesn't have an .sh file included > > What cannot you understand in the Installing section of README.md? > > ? sudo python -m pip install --upgrade pip setuptools wheel > ? sudo pip install Adafruit-SSD1306 > > ? Or alternatively: > > ? sudo python -m pip install --upgrade pip setuptools wheel > ? git clone https://github.com/adafruit/Adafruit_Python_SSD1306.git > ? cd Adafruit_Python_SSD1306 > ? sudo python setup.py install > > even WORSE, what cannot you undertand at the top of same file? > > ? This library has been deprecated! We are leaving this up for > ? historical and research purposes but archiving the repository. > > I was looking at the wrong file previously, and got mixed up, doh! I have installed the Adafruit_Python_SSD1306 library now. (There is no mention that I can see about installing other libraries etc to get the project to work, by the author) I had to make some changes in the authors file here: import time import Adafruit_SSD1306 import RPi.GPIO as GPIO <<< disp.begin() File "build/bdist.linux-armv6l/egg/Adafruit_SSD1306/SSD1306.py", line 148, in begin File "build/bdist.linux-armv6l/egg/Adafruit_SSD1306/SSD1306.py", line 247, in _initialize File "build/bdist.linux-armv6l/egg/Adafruit_SSD1306/SSD1306.py", line 129, in command File "build/bdist.linux-armv6l/egg/Adafruit_GPIO/I2C.py", line 116, in write8 File "build/bdist.linux-armv6l/egg/Adafruit_PureIO/smbus.py", line 268, in write_byte_data IOError: [Errno 121] Remote I/O error. Maybe I have created these errors unknowingly by the said changes I made. Thanks From python at mrabarnett.plus.com Thu Dec 5 09:17:07 2019 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 5 Dec 2019 14:17:07 +0000 Subject: What does the blue color section in background mean? In-Reply-To: References: Message-ID: <5982de96-8609-eee2-9b94-cbbd8dabb874@mrabarnett.plus.com> On 2019-12-05 03:28, lampahome wrote: > I tried to plot graph about a time-series with library statsmodel. > > I decide to plot autocorrelation function, but I don't know the blue > section in the example graph mean... > > Can anyone tell me? > > The plot_acf example link: > https://www.statsmodels.org/dev/generated/statsmodels.graphics.tsaplots.plot_acf.html#statsmodels.graphics.tsaplots.plot_acf > > > The graph in example: > https://www.statsmodels.org/dev/plots/graphics_tsa_plot_acf.png > I think that's the confidence interval. Here's another example that shows the same kind of thing: https://machinelearningmastery.com/gentle-introduction-autocorrelation-partial-autocorrelation/ From aishan0403 at gmail.com Thu Dec 5 10:31:54 2019 From: aishan0403 at gmail.com (A S) Date: Thu, 5 Dec 2019 07:31:54 -0800 (PST) Subject: Extract sentences in nested parentheses using Python In-Reply-To: References: <7a365fa0-e721-4a95-8c30-d3661301a7b2@googlegroups.com> Message-ID: On Tuesday, 3 December 2019 23:48:21 UTC+8, Peter Otten wrote: > A S wrote: > > > On Tuesday, 3 December 2019 01:01:25 UTC+8, Peter Otten wrote: > >> A S wrote: > >> > >> I think I've seen this question before ;) > >> > >> > I am trying to extract all strings in nested parentheses (along with > >> > the parentheses itself) in my .txt file. Please see the sample .txt > >> > file that I have used in this example here: > >> > (https://drive.google.com/open?id=1UKc0ZgY9Fsz5O1rSeBCLqt5dwZkMaQgr). > >> > > >> > I have tried and done up three different codes but none of them seems > >> > to be able to extract all the nested parentheses. They can only extract > >> > a portion of the nested parentheses. Any advice on what I've done wrong > >> > could really help! > >> > > >> > Here are the three codes I have done so far: > >> > > >> > 1st attempt: > >> > > >> > import re > >> > from os.path import join > >> > > >> > def balanced_braces(args): > >> > parts = [] > >> > for arg in args: > >> > if '(' not in arg: > >> > continue > >> > >> There could still be a ")" that you miss > >> > >> > chars = [] > >> > n = 0 > >> > for c in arg: > >> > if c == '(': > >> > if n > 0: > >> > chars.append(c) > >> > n += 1 > >> > elif c == ')': > >> > n -= 1 > >> > if n > 0: > >> > chars.append(c) > >> > elif n == 0: > >> > parts.append(''.join(chars).lstrip().rstrip()) > >> > chars = [] > >> > elif n > 0: > >> > chars.append(c) > >> > return parts > >> > >> It's probably easier to understand and implement when you process the > >> complete text at once. Then arbitrary splits don't get in the way of your > >> quest for ( and ). You just have to remember the position of the first > >> opening ( and number of opening parens that have to be closed before you > >> take the complete expression: > >> > >> level: 00011112222100 > >> text: abc(def(gh))ij > >> when we are here^ > >> we need^ > >> > >> A tentative implementation: > >> > >> $ cat parse.py > >> import re > >> > >> NOT_SET = object() > >> > >> def scan(text): > >> level = 0 > >> start = NOT_SET > >> for m in re.compile("[()]").finditer(text): > >> if m.group() == ")": > >> level -= 1 > >> if level < 0: > >> raise ValueError("underflow: more closing than opening > >> parens") > >> if level == 0: > >> # outermost closing parenthesis: > >> # deliver enclosed string including parens. > >> yield text[start:m.end()] > >> start = NOT_SET > >> elif m.group() == "(": > >> if level == 0: > >> # outermost opening parenthesis: remember position. > >> assert start is NOT_SET > >> start = m.start() > >> level += 1 > >> else: > >> assert False > >> if level > 0: > >> raise ValueError("unclosed parens remain") > >> > >> > >> if __name__ == "__main__": > >> with open("lan sample text file.txt") as instream: > >> text = instream.read() > >> for chunk in scan(text): > >> print(chunk) > >> $ python3 parse.py > >> ("xE'", PUT(xx.xxxx.),"'") > >> ("TRUuuuth") > > > > Hello Peter! I tried this on my actual working files and it returned this > > error: "unclosed parens remain". In this case, how can I continue to parse > > through my text files by only extracting those with balanced parentheses > > and ignore those that are incomplete? > > filenames = ... > for filename in filenames: > with open(filename) as instream: > text = instream.read() > try: > chunks = list(scan(text)) > except ValueError as err: > print(f"{err} in file {filename!r}", file=sys.stderr) > else: > for chunk in chunks: > print(chunk) hey Peter, it works! Thank you :) From aishan0403 at gmail.com Thu Dec 5 10:33:41 2019 From: aishan0403 at gmail.com (A S) Date: Thu, 5 Dec 2019 07:33:41 -0800 (PST) Subject: Extract sentences in nested parentheses using Python In-Reply-To: References: <7a365fa0-e721-4a95-8c30-d3661301a7b2@googlegroups.com> <1d689634-c3db-dd3f-c432-111604d5a784@DancesWithMice.info> Message-ID: <9ea51f20-4ed7-4eaa-9042-8aa9d4d2eb6a@googlegroups.com> On Tuesday, 3 December 2019 06:22:52 UTC+8, DL Neil wrote: > On 3/12/19 6:00 AM, Peter Otten wrote: > > A S wrote: > > I think I've seen this question before ;) > > In addition to 'other reasons' for @Peter's comment, it is a common > ComSc worked-problem or assignment. (in which case, we'd appreciate > being told that you/OP is asking for help with "homework") > > > >> I am trying to extract all strings in nested parentheses (along with the > >> parentheses itself) in my .txt file. Please see the sample .txt file that > >> I have used in this example here: > >> (https://drive.google.com/open?id=1UKc0ZgY9Fsz5O1rSeBCLqt5dwZkMaQgr). > >> > >> I have tried and done up three different codes but none of them seems to > >> be able to extract all the nested parentheses. They can only extract a > >> portion of the nested parentheses. Any advice on what I've done wrong > >> could really help! > > One approach is to research in the hope that there are already existing > tools or techniques which may help/save you from 'reinventing the wheel' > - when you think about it, a re-statement of open-source objectives. > > How does the Python interpreter break-down Python (text) code into its > constituent parts ("tokens") *including* parentheses? Such are made > available in (perhaps) a lesser-known corner of the PSL (Python Standard > Library). Might you be able to use one such tool? > > The ComSc technique which sprang to mind involves "stacks" (a LIFO data > structure) and "RPN" (Reverse Polish Notation). Whereas we like people > to take their turn when it comes to limited resources, eg to form a > "queue" to purchase/pay for goods at the local store, which is "FIFO" > (first-in, first-out); a "stack"/LIFO (last-in, first-out) can be > problematic to put into practical application. There are plenty of > Python implementations or you can 'roll your own' with a list. Again, > I'd likely employ a "deque" from the PSL's Collections library (although > as a "stack" rather than as a "double-ended queue"), because the > optimisation comes "free". (to my laziness, but after some kind soul > sweated-bullets to make it fast (in both senses) for 'the rest of us'!) > > > > It's probably easier to understand and implement when you process the > > complete text at once. Then arbitrary splits don't get in the way of your > > quest for ( and ). You just have to remember the position of the first > > opening ( and number of opening parens that have to be closed before you > > take the complete expression: > > +1 > but as a 'silver surfer', I don't like to be asked to "remember" anything! > > > > level: 00011112222100 > > we need^ > > > Consider: > original_text (the contents of the .txt file - add buffering if volumes > are huge) > current_text (the characters we have processed/"recognised" so-far) > stack (what an original name for such a data-structure! Which will > contain each of the partial parenthetical expressions found - but yet to > be proven/made complete) > > set current_text to NULSTRING > for each current_character in original_text: > if current_character is LEFT_PARENTHESIS: > push current_text to stack > set current_text to LEFT_PARENTHESIS > concatenate current_character with current_text > if current_character is RIGHT_PARENTHESIS: > # current_text is a parenthetical expression > # do with it what you will > pop the stack > set current_text to the ex-stack string \ > concat current_text's p-expn > > Once working: cover 'special cases' (after above loop), eg original_text > which doesn't begin and/or end with parentheses; and error cases, eg > pop-ping a NULSTRING, or thinking things are finished but the stack is > not yet empty - likely events from unbalanced parentheses! > > original text = "abc(def(gh))ij" > > event 1: in-turn, concatenate characters "abc" as current_text > event 2: locate (first) left-parenthesis, push current_text to stack(&) > event 3: concatenate "(def" > event 4: push, likewise > event 5: concatenate "(gh" > event 6: locate (first) right-parenthesis (matches to left-parenthesis > begining the current_string!) > result?: ?print current_text? > event 7: pop stack and redefine current_text as "(def(gh)" > event 8: repeat, per event 6 > event 9: current_text will now become "(def(gh))" > event 10: (special case) run-out of input at "(def(gh)ij" > event 11: (special case) pop (!any) stack and report "abc(def(gh)" > > > NB not sure of need for a "level" number; but if required, you can infer > that at any time, from the depth/len() of the stack! > > NBB being a simple-boy, my preference is for a 'single layer' of code, > cf @Peter's generator. Regardless the processes are "tokenisation" and > "recognition". > > At the back of my mind, was the notion that you may (eventually) be > required to work with more than parentheses, eg pair-wise > square-brackets and/or quotation-marks. In which case, you will need to > also 'push' the token and check for token-pairs when 'pop-ping', as well > as (perhaps) recognising lists of tokens to tokenise instead of the two > parenthesis characters alone. In which case, I'd take a serious look at > the Python Language Services rather than taking a 'roll your own' approach! > > Contrarily, if this spec is 'it', then you might consider optimising the > search processes which 'trigger' the two stack operations, by re-working > the for-loop and utilising string.find() - prioritising whichever > parenthesis is left-most/comes-first - assuming LtR text. (apologies if > you have already tried this in one of your previous approaches) > Unfortunately, such likely results in 'layers' of code, and a generator > might well become the tool-of-choice (I say this before @Peter comes > back and (quite deservedly) flays me alive!). > > > WebRefs: > Python Language Services: https://docs.python.org/3/library/language.html > collections ? Container datatypes: > https://docs.python.org/3/library/collections.html > > See also your ComSc text/reference materials. > -- > Regards =dn Hey DL Neil, this is rather sophisticated for me as I am still learning the basics of Python...But I truly appreciate your help and effort! I did try to read through what you said, but some parts I could not register! From rob at despammer.com Thu Dec 5 13:49:40 2019 From: rob at despammer.com (RobH) Date: Thu, 5 Dec 2019 18:49:40 +0000 Subject: ImportError: No module named Adafruit_SSD1306 Update In-Reply-To: References: <5de83dea$0$31397$426a74cc@news.free.fr> Message-ID: On 05/12/2019 10:07, RobH wrote: > On 04/12/2019 23:15, Python wrote: >> Le 05/12/2019 ? 00:06, RobH a ?crit?: >>> On 04/12/2019 22:33, Wildman wrote: >>>> On Wed, 04 Dec 2019 20:25:33 +0000, RobH wrote: >>>> >>>>> I am trying to do this project on a pi zero: >>>>> >>>>> http://frederickvandenbosch.be/?p=1365 >>>>> >>>>> I copied the code to the pi zero Download folder and when I run it >>>>> I get >>>>> the above error at line 4 >>>>> Import Adafruit_SSD1306 >>>>> >>>>> I am using python version 2.7.16, if that makes any difference >>>>> I have the same module as the authors' link goes to : >>>>> >>>>> Monochrome 1.3" 128x64 OLED graphic display - STEMMA QT / Qwiic >>>>> >>>>> Have I missed something. >>>> >>>> The error indicates that Adafruit_SSD1306 in not installed. >>>> >>>> https://github.com/adafruit/Adafruit_Python_SSD1306 >>>> >>> >>> I have the library in the same Downloads folder, but I don't know how >>> to actually install it as it doesn't have an .sh file included >> >> What cannot you understand in the Installing section of README.md? >> >> ?? sudo python -m pip install --upgrade pip setuptools wheel >> ?? sudo pip install Adafruit-SSD1306 >> >> ?? Or alternatively: >> >> ?? sudo python -m pip install --upgrade pip setuptools wheel >> ?? git clone https://github.com/adafruit/Adafruit_Python_SSD1306.git >> ?? cd Adafruit_Python_SSD1306 >> ?? sudo python setup.py install >> >> even WORSE, what cannot you undertand at the top of same file? >> >> ?? This library has been deprecated! We are leaving this up for >> ?? historical and research purposes but archiving the repository. >> >> > > I was looking at the wrong file previously, and got mixed up, doh! > I have installed the Adafruit_Python_SSD1306 library now. > > (There is no mention that I can see about installing other libraries etc > to get the project to work, by the author) > Update: I did python3 Internet.py and now only get this error: pi at raspberrypi:~/Downloads $ python3 Internet.py File "Internet.py", line 24 font = ImageFont.truetype( 'Minecraftia.ttf', 35) ^ TabError: inconsistent use of tabs and spaces in indentation I cannot see what is wrong, as the text is all lined up with that above and below: def display_time(): # Collect current time and date if(time_format): current_time = time.strftime("%I:%M") else: current_time = time.strftime("%H:%M") current_date = time.strftime("%d/%m/%Y") # Clear image buffer by drawing a black filled box draw.rectangle((0,0,width,height), outline=0, fill=0) # Set font type and size font = ImageFont.truetype ('Minecraftia.ttf', 35) << error here # Position time x_pos = (disp.width/2)-(string_width(font,current_time)/2) y_pos = 2 + (disp.height-4-8)/2 - (35/2) Thanks From python at mrabarnett.plus.com Thu Dec 5 14:28:43 2019 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 5 Dec 2019 19:28:43 +0000 Subject: ImportError: No module named Adafruit_SSD1306 Update In-Reply-To: References: <5de83dea$0$31397$426a74cc@news.free.fr> Message-ID: On 2019-12-05 18:49, RobH wrote: > On 05/12/2019 10:07, RobH wrote: >> On 04/12/2019 23:15, Python wrote: >>> Le 05/12/2019 ? 00:06, RobH a ?crit?: >>>> On 04/12/2019 22:33, Wildman wrote: >>>>> On Wed, 04 Dec 2019 20:25:33 +0000, RobH wrote: >>>>> >>>>>> I am trying to do this project on a pi zero: >>>>>> >>>>>> http://frederickvandenbosch.be/?p=1365 >>>>>> >>>>>> I copied the code to the pi zero Download folder and when I run it >>>>>> I get >>>>>> the above error at line 4 >>>>>> Import Adafruit_SSD1306 >>>>>> >>>>>> I am using python version 2.7.16, if that makes any difference >>>>>> I have the same module as the authors' link goes to : >>>>>> >>>>>> Monochrome 1.3" 128x64 OLED graphic display - STEMMA QT / Qwiic >>>>>> >>>>>> Have I missed something. >>>>> >>>>> The error indicates that Adafruit_SSD1306 in not installed. >>>>> >>>>> https://github.com/adafruit/Adafruit_Python_SSD1306 >>>>> >>>> >>>> I have the library in the same Downloads folder, but I don't know how >>>> to actually install it as it doesn't have an .sh file included >>> >>> What cannot you understand in the Installing section of README.md? >>> >>> ?? sudo python -m pip install --upgrade pip setuptools wheel >>> ?? sudo pip install Adafruit-SSD1306 >>> >>> ?? Or alternatively: >>> >>> ?? sudo python -m pip install --upgrade pip setuptools wheel >>> ?? git clone https://github.com/adafruit/Adafruit_Python_SSD1306.git >>> ?? cd Adafruit_Python_SSD1306 >>> ?? sudo python setup.py install >>> >>> even WORSE, what cannot you undertand at the top of same file? >>> >>> ?? This library has been deprecated! We are leaving this up for >>> ?? historical and research purposes but archiving the repository. >>> >>> >> >> I was looking at the wrong file previously, and got mixed up, doh! >> I have installed the Adafruit_Python_SSD1306 library now. >> >> (There is no mention that I can see about installing other libraries etc >> to get the project to work, by the author) >> > > Update: > I did python3 Internet.py > and now only get this error: > > pi at raspberrypi:~/Downloads $ python3 Internet.py > File "Internet.py", line 24 > font = ImageFont.truetype( 'Minecraftia.ttf', 35) > ^ > TabError: inconsistent use of tabs and spaces in indentation > > I cannot see what is wrong, as the text is all lined up with that above > and below: > Are you sure that the indentation is the same? Do all of the lines use all spaces or all tabs or indentation? Just because they look lined-up doesn't mean that they're doing it with the sequence of characters. > def display_time(): > # Collect current time and date > if(time_format): > current_time = time.strftime("%I:%M") > else: > current_time = time.strftime("%H:%M") > > current_date = time.strftime("%d/%m/%Y") > > # Clear image buffer by drawing a black filled box > draw.rectangle((0,0,width,height), outline=0, fill=0) > > # Set font type and size > font = ImageFont.truetype ('Minecraftia.ttf', 35) << error here > > # Position time > x_pos = (disp.width/2)-(string_width(font,current_time)/2) > y_pos = 2 + (disp.height-4-8)/2 - (35/2) > > From rhodri at kynesim.co.uk Thu Dec 5 14:30:31 2019 From: rhodri at kynesim.co.uk (Rhodri James) Date: Thu, 5 Dec 2019 19:30:31 +0000 Subject: ImportError: No module named Adafruit_SSD1306 Update In-Reply-To: References: <5de83dea$0$31397$426a74cc@news.free.fr> Message-ID: <1c044f94-c429-b7d8-150c-7068148a841e@kynesim.co.uk> On 05/12/2019 18:49, RobH wrote: > Update: > I did python3 Internet.py > and now only get this error: > > pi at raspberrypi:~/Downloads $ python3 Internet.py > ? File "Internet.py", line 24 > ??? font = ImageFont.truetype( 'Minecraftia.ttf', 35) > ??????????????????????????????????????????????????? ^ > TabError: inconsistent use of tabs and spaces in indentation > > I cannot see what is wrong, as the text is all lined up with that above > and below: The problem will be that you have a mix of tabs and spaces in your indentation. This causes problems because some people don't think that the One True Tab Width is 8 characters ;-) so to them the indentation looks ragged. Worse, when they mix tabs and spaces, code that looks to be at the same indentation level to them looks different to the interpreter. The decision was taken a while ago that Python should put its foot down about this, and demand that we use either all tabs or all spaces for our indentation. That's what you've fallen foul off; there must be a mix of tabs and spaces in that line! -- Rhodri James *-* Kynesim Ltd From rhodri at kynesim.co.uk Thu Dec 5 14:40:16 2019 From: rhodri at kynesim.co.uk (Rhodri James) Date: Thu, 5 Dec 2019 19:40:16 +0000 Subject: ImportError: No module named Adafruit_SSD1306 Update In-Reply-To: <1c044f94-c429-b7d8-150c-7068148a841e@kynesim.co.uk> References: <5de83dea$0$31397$426a74cc@news.free.fr> <1c044f94-c429-b7d8-150c-7068148a841e@kynesim.co.uk> Message-ID: <6d5f872f-5f96-8b00-d6a9-ffab3d177de7@kynesim.co.uk> On 05/12/2019 19:30, Rhodri James wrote: > On 05/12/2019 18:49, RobH wrote: >> Update: >> I did python3 Internet.py >> and now only get this error: >> >> pi at raspberrypi:~/Downloads $ python3 Internet.py >> ?? File "Internet.py", line 24 >> ???? font = ImageFont.truetype( 'Minecraftia.ttf', 35) >> ???????????????????????????????????????????????????? ^ >> TabError: inconsistent use of tabs and spaces in indentation >> >> I cannot see what is wrong, as the text is all lined up with that >> above and below: > > The problem will be that you have a mix of tabs and spaces in your > indentation.? This causes problems because some people don't think that > the One True Tab Width is 8 characters ;-) so to them the indentation > looks ragged.? Worse, when they mix tabs and spaces, code that looks to > be at the same indentation level to them looks different to the > interpreter.? The decision was taken a while ago that Python should put > its foot down about this, and demand that we use either all tabs or all > spaces for our indentation.? That's what you've fallen foul off; there > must be a mix of tabs and spaces in that line! Or more likely you've used tabs on that line and spaces elsewhere, or vice versa. I should have remember to say that, sorry. -- Rhodri James *-* Kynesim Ltd From rob at despammer.com Thu Dec 5 15:28:52 2019 From: rob at despammer.com (RobH) Date: Thu, 5 Dec 2019 20:28:52 +0000 Subject: ImportError: No module named Adafruit_SSD1306 Update In-Reply-To: References: <5de83dea$0$31397$426a74cc@news.free.fr> <1c044f94-c429-b7d8-150c-7068148a841e@kynesim.co.uk> <6d5f872f-5f96-8b00-d6a9-ffab3d177de7@kynesim.co.uk> Message-ID: On 05/12/2019 19:40, Rhodri James wrote: > On 05/12/2019 19:30, Rhodri James wrote: >> On 05/12/2019 18:49, RobH wrote: >>> Update: >>> I did python3 Internet.py >>> and now only get this error: >>> >>> pi at raspberrypi:~/Downloads $ python3 Internet.py >>> ?? File "Internet.py", line 24 >>> ???? font = ImageFont.truetype( 'Minecraftia.ttf', 35) >>> ???????????????????????????????????????????????????? ^ >>> TabError: inconsistent use of tabs and spaces in indentation >>> >>> I cannot see what is wrong, as the text is all lined up with that >>> above and below: >> >> The problem will be that you have a mix of tabs and spaces in your >> indentation.? This causes problems because some people don't think >> that the One True Tab Width is 8 characters ;-) so to them the >> indentation looks ragged.? Worse, when they mix tabs and spaces, code >> that looks to be at the same indentation level to them looks different >> to the interpreter.? The decision was taken a while ago that Python >> should put its foot down about this, and demand that we use either all >> tabs or all spaces for our indentation.? That's what you've fallen >> foul off; there must be a mix of tabs and spaces in that line! > > Or more likely you've used tabs on that line and spaces elsewhere, or > vice versa.? I should have remember to say that, sorry. > Ok thanks for the explanation there, and I have placed the cursor at the beginning of the first indented line. Moving down 1 line at a time , each line is at the same position upto line 157 in the authors code . Then it is closer in to the edge upto line 190, where it goes back out again. What is my best course of action here now. From joel.goldstick at gmail.com Thu Dec 5 15:39:45 2019 From: joel.goldstick at gmail.com (Joel Goldstick) Date: Thu, 5 Dec 2019 15:39:45 -0500 Subject: ImportError: No module named Adafruit_SSD1306 Update In-Reply-To: References: <5de83dea$0$31397$426a74cc@news.free.fr> <1c044f94-c429-b7d8-150c-7068148a841e@kynesim.co.uk> <6d5f872f-5f96-8b00-d6a9-ffab3d177de7@kynesim.co.uk> Message-ID: On Thu, Dec 5, 2019 at 3:31 PM RobH wrote: > > On 05/12/2019 19:40, Rhodri James wrote: > > On 05/12/2019 19:30, Rhodri James wrote: > >> On 05/12/2019 18:49, RobH wrote: > >>> Update: > >>> I did python3 Internet.py > >>> and now only get this error: > >>> > >>> pi at raspberrypi:~/Downloads $ python3 Internet.py > >>> File "Internet.py", line 24 > >>> font = ImageFont.truetype( 'Minecraftia.ttf', 35) > >>> ^ > >>> TabError: inconsistent use of tabs and spaces in indentation > >>> > >>> I cannot see what is wrong, as the text is all lined up with that > >>> above and below: > >> > >> The problem will be that you have a mix of tabs and spaces in your > >> indentation. This causes problems because some people don't think > >> that the One True Tab Width is 8 characters ;-) so to them the > >> indentation looks ragged. Worse, when they mix tabs and spaces, code > >> that looks to be at the same indentation level to them looks different > >> to the interpreter. The decision was taken a while ago that Python > >> should put its foot down about this, and demand that we use either all > >> tabs or all spaces for our indentation. That's what you've fallen > >> foul off; there must be a mix of tabs and spaces in that line! > > > > Or more likely you've used tabs on that line and spaces elsewhere, or > > vice versa. I should have remember to say that, sorry. > > > > Ok thanks for the explanation there, and I have placed the cursor at the > beginning of the first indented line. Moving down 1 line at a time , > each line is at the same position upto line 157 in the authors code . > Then it is closer in to the edge upto line 190, where it goes back out > again. > > What is my best course of action here now. > > -- > https://mail.python.org/mailman/listinfo/python-list google or duckduckgo or whatever your text editor and tabs to spaces.. there is probably an easy way to convert the file -- Joel Goldstick http://joelgoldstick.com/blog http://cc-baseballstats.info/stats/birthdays From python at mrabarnett.plus.com Thu Dec 5 15:55:42 2019 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 5 Dec 2019 20:55:42 +0000 Subject: ImportError: No module named Adafruit_SSD1306 Update In-Reply-To: References: <5de83dea$0$31397$426a74cc@news.free.fr> <1c044f94-c429-b7d8-150c-7068148a841e@kynesim.co.uk> <6d5f872f-5f96-8b00-d6a9-ffab3d177de7@kynesim.co.uk> Message-ID: On 2019-12-05 20:39, Joel Goldstick wrote: > On Thu, Dec 5, 2019 at 3:31 PM RobH wrote: >> >> On 05/12/2019 19:40, Rhodri James wrote: >> > On 05/12/2019 19:30, Rhodri James wrote: >> >> On 05/12/2019 18:49, RobH wrote: >> >>> Update: >> >>> I did python3 Internet.py >> >>> and now only get this error: >> >>> >> >>> pi at raspberrypi:~/Downloads $ python3 Internet.py >> >>> File "Internet.py", line 24 >> >>> font = ImageFont.truetype( 'Minecraftia.ttf', 35) >> >>> ^ >> >>> TabError: inconsistent use of tabs and spaces in indentation >> >>> >> >>> I cannot see what is wrong, as the text is all lined up with that >> >>> above and below: >> >> >> >> The problem will be that you have a mix of tabs and spaces in your >> >> indentation. This causes problems because some people don't think >> >> that the One True Tab Width is 8 characters ;-) so to them the >> >> indentation looks ragged. Worse, when they mix tabs and spaces, code >> >> that looks to be at the same indentation level to them looks different >> >> to the interpreter. The decision was taken a while ago that Python >> >> should put its foot down about this, and demand that we use either all >> >> tabs or all spaces for our indentation. That's what you've fallen >> >> foul off; there must be a mix of tabs and spaces in that line! >> > >> > Or more likely you've used tabs on that line and spaces elsewhere, or >> > vice versa. I should have remember to say that, sorry. >> > >> >> Ok thanks for the explanation there, and I have placed the cursor at the >> beginning of the first indented line. Moving down 1 line at a time , >> each line is at the same position upto line 157 in the authors code . >> Then it is closer in to the edge upto line 190, where it goes back out >> again. >> >> What is my best course of action here now. >> > google or duckduckgo or whatever your text editor and tabs to spaces.. > there is probably an easy way to convert the file > Or expand the tabs using Python: at the Python prompt, read it in, use the expandtabs method of str, write it back out. From tjreedy at udel.edu Thu Dec 5 18:11:30 2019 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 5 Dec 2019 18:11:30 -0500 Subject: ImportError: No module named Adafruit_SSD1306 Update In-Reply-To: References: <5de83dea$0$31397$426a74cc@news.free.fr> <1c044f94-c429-b7d8-150c-7068148a841e@kynesim.co.uk> <6d5f872f-5f96-8b00-d6a9-ffab3d177de7@kynesim.co.uk> Message-ID: On 12/5/2019 3:55 PM, MRAB wrote: >>> Ok thanks for the explanation there, and I have placed the cursor at the >>> beginning of the first indented line. Moving down 1 line at a time , >>> each line is at the same position upto line 157 in the authors code . >>> Then it is closer in to the edge upto line 190, where it goes back out >>> again. >>> >>> What is my best course of action here now. >>> >> google or duckduckgo or whatever your text editor and tabs to spaces.. >> there is probably an easy way to convert the file >> > Or expand the tabs using Python: at the Python prompt, read it in, use > the expandtabs method of str, write it back out. IDLE's Format menu has an option to do this in memory, so you can check before writing back to disk. -- Terry Jan Reedy From skip.montanaro at gmail.com Fri Dec 6 10:55:20 2019 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Fri, 6 Dec 2019 09:55:20 -0600 Subject: IPC10 archive anywhere? Message-ID: I've poked around a bit and have been unable to come up with an archive of the papers delivered at IPC10 (2001, I believe - pre-PyCon days). Might anyone have a link? Thanks, Skip From skip.montanaro at gmail.com Fri Dec 6 11:08:39 2019 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Fri, 6 Dec 2019 10:08:39 -0600 Subject: IPC10 archive anywhere? In-Reply-To: References: Message-ID: > I've poked around a bit and have been unable to come up with an > archive of the papers delivered at IPC10 (2001, I believe - pre-PyCon > days). Might anyone have a link? Found it a few minutes later: https://legacy.python.org/workshops/ Skip From arj.python at gmail.com Fri Dec 6 12:49:20 2019 From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer) Date: Fri, 6 Dec 2019 21:49:20 +0400 Subject: Zipapp can't find sqlite db Message-ID: Greetings, I'm using zipapp to include a gui + db __main__.py dbs/ file.db When packaging, the db is there. When querying through sqlalchemy, it says can't open db file. Help appreciated! Yours, Abdur-Rahmaan Janhangeer pythonmembers.club | github Mauritius From bob at mellowood.ca Fri Dec 6 13:17:48 2019 From: bob at mellowood.ca (Bob van der Poel) Date: Fri, 6 Dec 2019 11:17:48 -0700 Subject: Unicode filenames Message-ID: I have some files which came off the net with, I'm assuming, unicode characters in the names. I have a very short program which takes the filename and puts into an emacs buffer, and then lets me add information to that new file (it's a poor man's DB). Next, I can look up text in the file and open the saved filename. Everything works great until I hit those darn unicode filenames. Just to confuse me even more, the error seems to be coming from a bit of tkinter code: if sresults.has_key(textAtCursor): bookname = os.path.expanduser(sresults[textAtCursor].strip()) which generates UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal if sresults.has_key(textAtCursor): I really don't understand the business about "both arguments". Not sure how to proceed here. Hoping for a guideline! Thanks. -- **** Listen to my FREE CD at http://www.mellowood.ca/music/cedars **** Bob van der Poel ** Wynndel, British Columbia, CANADA ** EMAIL: bob at mellowood.ca WWW: http://www.mellowood.ca From PythonList at danceswithmice.info Fri Dec 6 14:40:47 2019 From: PythonList at danceswithmice.info (DL Neil) Date: Sat, 7 Dec 2019 08:40:47 +1300 Subject: Unicode filenames In-Reply-To: References: Message-ID: <2bb4a9d3-4941-1ab5-5acc-acbde6b57e34@DancesWithMice.info> On 7/12/19 7:17 AM, Bob van der Poel wrote: > I have some files which came off the net with, I'm assuming, unicode > characters in the names. I have a very short program which takes the > filename and puts into an emacs buffer, and then lets me add information to > that new file (it's a poor man's DB). > > Next, I can look up text in the file and open the saved filename. > Everything works great until I hit those darn unicode filenames. > > Just to confuse me even more, the error seems to be coming from a bit of > tkinter code: > if sresults.has_key(textAtCursor): > bookname = os.path.expanduser(sresults[textAtCursor].strip()) > > which generates > > UnicodeWarning: Unicode equal comparison failed to convert both arguments > to Unicode - interpreting them as being unequal if > sresults.has_key(textAtCursor): > > I really don't understand the business about "both arguments". Not sure how > to proceed here. Hoping for a guideline! (I'm guessing that) the "both arguments" relates to expanduser() because this is the first time that the fileNM has been identified to Python as anything more than a string of characters. [a fileNM will be a string of characters, but a string of characters is not necessarily a (legal) fileNM!] Further suggesting, that if you are using Python3 (cf 2), your analysis may be the wrong-way-around. Python3 treats strings as Unicode. However, there is, and certainly in the past, was, no requirement for OpSys and IOCS to encode in Unicode. The problem (for me) came from MSFT's (for example) many variations of ISO-8859-n and that there are no clues as to which of these was used in naming the file, and thus many possibly 'translations' into Unicode. You can start to address the issue by using Python's bytes (instead of strings), however that cold reality still intrudes. Do you know the provenance of these files, eg they are in French and from an MS-Win machine? If so, you may be able to use decode() and encode(), but... Still looking for trouble? Knowing a fileNM was in Spanish/Portuguese I was able to take the fileNM's individual Unicode characters/surrogates and subtract an applicable constant, so that accented letters fell 'back' into the correct Unicode range. (this is extremely risky, and could quite easily make matters worse/more confusing). I warn you that pursuing this matter involves disappearing down into a very deep 'rabbit hole', but YMMV! WebRefs: https://docs.python.org/3/howto/unicode.html https://www.dictionary.com/e/slang/rabbit-hole/ -- Regards =dn From ijbrewster at alaska.edu Fri Dec 6 15:58:36 2019 From: ijbrewster at alaska.edu (Israel Brewster) Date: Fri, 6 Dec 2019 11:58:36 -0900 Subject: Make warning an exception? Message-ID: <1EED26AD-2A95-42B0-8D31-58D9854D185B@alaska.edu> I was running some code and I saw this pop up in the console: 2019-12-06 11:53:54.087 Python[85524:39651849] WARNING: nextEventMatchingMask should only be called from the Main Thread! This will throw an exception in the future. The only problem is, I have no idea what is generating that warning - I never call nextEventMatchingMask directly, so it must be getting called from one of the libraries I?m calling. Is there some way I can force python to throw an exception now, so my debugger can catch it and let me know where in my code the originating call is? I?ve tried stepping through the obvious options, with no luck so far. --- Israel Brewster Software Engineer Alaska Volcano Observatory Geophysical Institute - UAF 2156 Koyukuk Drive Fairbanks AK 99775-7320 Work: 907-474-5172 cell: 907-328-9145 From rgaddi at highlandtechnology.invalid Fri Dec 6 16:11:27 2019 From: rgaddi at highlandtechnology.invalid (Rob Gaddi) Date: Fri, 6 Dec 2019 13:11:27 -0800 Subject: Make warning an exception? In-Reply-To: References: <1EED26AD-2A95-42B0-8D31-58D9854D185B@alaska.edu> Message-ID: On 12/6/19 12:58 PM, Israel Brewster wrote: > I was running some code and I saw this pop up in the console: > > 2019-12-06 11:53:54.087 Python[85524:39651849] WARNING: nextEventMatchingMask should only be called from the Main Thread! This will throw an exception in the future. > > The only problem is, I have no idea what is generating that warning - I never call nextEventMatchingMask directly, so it must be getting called from one of the libraries I?m calling. Is there some way I can force python to throw an exception now, so my debugger can catch it and let me know where in my code the originating call is? I?ve tried stepping through the obvious options, with no luck so far. > > --- > Israel Brewster > Software Engineer > Alaska Volcano Observatory > Geophysical Institute - UAF > 2156 Koyukuk Drive > Fairbanks AK 99775-7320 > Work: 907-474-5172 > cell: 907-328-9145 > You need to set the warning filter to "error", which you can do either with warnings.simplefilter at the start of your program or by setting the PYTHONWARNINGS environment variable. https://docs.python.org/3/library/warnings.html#the-warnings-filter This the same project you're having PySide/threading problems on? From PythonList at danceswithmice.info Fri Dec 6 16:16:13 2019 From: PythonList at danceswithmice.info (DL Neil) Date: Sat, 7 Dec 2019 10:16:13 +1300 Subject: Make warning an exception? In-Reply-To: <1EED26AD-2A95-42B0-8D31-58D9854D185B@alaska.edu> References: <1EED26AD-2A95-42B0-8D31-58D9854D185B@alaska.edu> Message-ID: <9a617d11-acc1-f810-2c7c-72d7b8b067b6@DancesWithMice.info> On 7/12/19 9:58 AM, Israel Brewster wrote: > I was running some code and I saw this pop up in the console: > > 2019-12-06 11:53:54.087 Python[85524:39651849] WARNING: nextEventMatchingMask should only be called from the Main Thread! This will throw an exception in the future. > > The only problem is, I have no idea what is generating that warning - I never call nextEventMatchingMask directly, so it must be getting called from one of the libraries I?m calling. Is there some way I can force python to throw an exception now, so my debugger can catch it and let me know where in my code the originating call is? I?ve tried stepping through the obvious options, with no luck so far. We are able to "filter" errors, including turning warnings into full-bore errors. Of possible use: https://docs.python.org/3/library/warnings.html -- Regards =dn From tjreedy at udel.edu Fri Dec 6 18:20:26 2019 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 6 Dec 2019 18:20:26 -0500 Subject: Unicode filenames In-Reply-To: References: Message-ID: On 12/6/2019 1:17 PM, Bob van der Poel wrote: > I have some files which came off the net with, I'm assuming, unicode > characters in the names. I have a very short program which takes the > filename and puts into an emacs buffer, and then lets me add information to > that new file (it's a poor man's DB). > > Next, I can look up text in the file and open the saved filename. > Everything works great until I hit those darn unicode filenames. > > Just to confuse me even more, the error seems to be coming from a bit of > tkinter code: > if sresults.has_key(textAtCursor): > bookname = os.path.expanduser(sresults[textAtCursor].strip()) 'textAtCursor' does not appear in any 3.9 tkinter/*.py file > which generates > > UnicodeWarning: Unicode equal comparison failed to convert both arguments > to Unicode - interpreting them as being unequal if > sresults.has_key(textAtCursor): > > I really don't understand the business about "both arguments". 'sresults.has_key(textAtCursor)' will see if the hash value of textAtCursor matches the hash value of any key and then compare the strings. 'failed to convert' suggests to me that you are running 2.x and that one of the strings is bytes and the other unicode. Not sure how > to proceed here. Hoping for a guideline! > > Thanks. > > -- Terry Jan Reedy From spayth77 at gmail.com Fri Dec 6 18:53:11 2019 From: spayth77 at gmail.com (Sam Paython) Date: Fri, 6 Dec 2019 15:53:11 -0800 (PST) Subject: Error getting data from website Message-ID: <789b0fa9-e2ef-4daa-9e85-d3c8b34223cd@googlegroups.com> Hi all, This is the code I am writing: import requests from bs4 import BeautifulSoup request = requests.get("https://www.amazon.ca/dp/B07RZFQ6HC") content = request.content soup = BeautifulSoup(content, "html.parser") element = soup.find("span",{"id":"priceblock_dealprice"}) print(element.text.strip()) and this is the error I am getting: C:\Users\Sam\PycharmProjects\untitled2\venv\Scripts\python.exe C:/Users/Sam/PycharmProjects/untitled2/src/app.py Traceback (most recent call last): File "C:/Users/Sam/PycharmProjects/untitled2/src/app.py", line 9, in print(element.text.strip()) AttributeError: 'NoneType' object has no attribute 'text' Could someone please help? From PythonList at DancesWithMice.info Fri Dec 6 19:31:05 2019 From: PythonList at DancesWithMice.info (DL Neil) Date: Sat, 7 Dec 2019 13:31:05 +1300 Subject: Error getting data from website In-Reply-To: <789b0fa9-e2ef-4daa-9e85-d3c8b34223cd@googlegroups.com> References: <789b0fa9-e2ef-4daa-9e85-d3c8b34223cd@googlegroups.com> Message-ID: <0dfb757a-0549-4c15-0e67-48511e10f671@DancesWithMice.info> On 7/12/19 12:53 PM, Sam Paython wrote: > This is the code I am writing: > import requests > from bs4 import BeautifulSoup > request = requests.get("https://www.amazon.ca/dp/B07RZFQ6HC") > content = request.content > soup = BeautifulSoup(content, "html.parser") > element = soup.find("span",{"id":"priceblock_dealprice"}) > print(element.text.strip()) > > and this is the error I am getting: > C:\Users\Sam\PycharmProjects\untitled2\venv\Scripts\python.exe C:/Users/Sam/PycharmProjects/untitled2/src/app.py > Traceback (most recent call last): > File "C:/Users/Sam/PycharmProjects/untitled2/src/app.py", line 9, in > print(element.text.strip()) > AttributeError: 'NoneType' object has no attribute 'text' > > Could someone please help? The err.msg/stack-trace is your friend! The comment about "NoneType" means 'there's nothing there' (roughly!) to print(). The question then becomes: "why?" or "why not?"... With a short piece of code like this, and (I am assuming) trying-out a library for the first time, may I recommend that you use the Python REPL, because it allows you to 'see' what's going-on behind the scenes/underneath the hood - and ultimately, reveals the problem. From a Python terminal (cmd is appropriate to your PC's OpSys): [dn at JrBrown ~]$ python3 Python 3.7.4 (default, Jul 9 2019, 16:48:28) [GCC 8.3.1 20190223 (Red Hat 8.3.1-2)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import requests >>> from bs4 import BeautifulSoup >>> request = requests.get("https://www.amazon.ca/dp/B07RZFQ6HC") >>> request # notice how I'm asking to 'see' what happened >>> content = request.content >>> content # there is no need to enclose in print()! b'\n 1 np.dtype([('Q', 'I', 'Q')]) > > ValueError: mismatch in size of old and new data-descriptor > > In [3]: np.dtype([('field1', 'Q'), ('field2', 'I'), ('field3', > 'Q')])??????????????????????????????????????????????????????????????? > Out[3]: dtype([('field1', ' > In [4]:??? > > ... and now let's put it all together! s1 = struct.Struct("@QIQ") ss1 = s1.pack(1,11,111) struct_dtype = np.dtype([('field1', 'Q'), ('field2', 'I'), ('field3', 'Q')]) a = np.frombuffer(ss1, dtype=struct_dtype) I'm using the frombuffer() function deliberately so I don't have to figure out the shape of the final array (which is (1,), not (3,), by the way). And hey presto: it raises an exception! > ValueError: buffer size must be a multiple of element size Your example shows a difference between the default behaviour of numpy's structured dtype and the struct module: packing! By default, numpy structured dtypes are closely packed, i.e. nothing is aligned to useful memory boundaries. struct_type.itemsize == 20 The struct module, on the other hand, tries to guess where the C compiler would put its padding. len(ss1) == 24 We can tell numpy to do the same: struct_dtype = np.dtype([('field1', 'Q'), ('field2', 'I'), ('field3', 'Q')], align=True) and then a = np.frombuffer(ss1, dtype=struct_dtype) works and produces array([(1, 11, 111)], ????? dtype={'names':['field1','field2','field3'], 'formats':[' References: <303a8c7b-1c1b-12c0-bb0c-4cd70be8611a@kynesim.co.uk> Message-ID: <6ff313c1-d38b-5c5b-99cc-743413416f2f@kynesim.co.uk> On 19/12/2019 11:23, tommy yama wrote: > Thanks for your kind response. > The error was simply "module Hexdump was not found". Several things: a) Did it really say "module Hexdump was not found"? "hexdump" and "Hexdump" are not the same things; module names are case-sensitive. b) There will have been a whole load more stack trace that might be useful to us. I should have been clearer when I asked you to copy and paste the error that I really meant the whole of the complaint that Python made to you, not just the final error message! Apologies for that. c) I would much prefer it if you didn't top-post, but interleaved your replies like I've done here. I find it hard to follow top-posted messages because they reverse the normal flow of conversation. > > > On Wed, Dec 18, 2019 at 11:39 PM Rhodri James wrote: > >> On 18/12/2019 02:23, tommy yama wrote: >>> Hi, >>> >>> This sounds familiar to somebody? >>> After upgrading my mac OS to Catalina, this persists even after pip3 >>> install hexdump. >>> >>> [image: image.png] >> >> I'm afraid this is a text-only mailing list. Your screenshot has been >> stripped out before any of us saw it. Could you copy and paste (DON'T >> retype!) the error instead, so we can all read it? >> >> -- >> Rhodri James *-* Kynesim Ltd >> -- >> https://mail.python.org/mailman/listinfo/python-list >> > -- Rhodri James *-* Kynesim Ltd From bluebox03 at gmail.com Thu Dec 19 07:43:11 2019 From: bluebox03 at gmail.com (tommy yama) Date: Thu, 19 Dec 2019 12:43:11 +0000 Subject: hexdump module installation error In-Reply-To: <6ff313c1-d38b-5c5b-99cc-743413416f2f@kynesim.co.uk> References: <303a8c7b-1c1b-12c0-bb0c-4cd70be8611a@kynesim.co.uk> <6ff313c1-d38b-5c5b-99cc-743413416f2f@kynesim.co.uk> Message-ID: Hi Rhodri, Thanks for your quick response i did not expect. I hope you see the error below in my response as i just copy and paste it. "no module named 'hexdump'." In addition, i tried to execute python3 hexdump.py. However, no such file directory. from hexdump import hexdump ModuleNotFoundError: No module named 'hexdump' user at USERnoMacBook-Air LibraBrowser % python3 hexdump.py /usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS/Python: can't open file 'hexdump.py': [Errno 2] No such file or directory user at USERnoMacBook-Air LibraBrowser % Cheers, On Thu, Dec 19, 2019 at 12:21 PM Rhodri James wrote: > On 19/12/2019 11:23, tommy yama wrote: > > Thanks for your kind response. > > The error was simply "module Hexdump was not found". > > Several things: > > a) Did it really say "module Hexdump was not found"? "hexdump" and > "Hexdump" are not the same things; module names are case-sensitive. > > b) There will have been a whole load more stack trace that might be > useful to us. I should have been clearer when I asked you to copy and > paste the error that I really meant the whole of the complaint that > Python made to you, not just the final error message! Apologies for that. > > c) I would much prefer it if you didn't top-post, but interleaved your > replies like I've done here. I find it hard to follow top-posted > messages because they reverse the normal flow of conversation. > > > > > > > On Wed, Dec 18, 2019 at 11:39 PM Rhodri James > wrote: > > > >> On 18/12/2019 02:23, tommy yama wrote: > >>> Hi, > >>> > >>> This sounds familiar to somebody? > >>> After upgrading my mac OS to Catalina, this persists even after pip3 > >>> install hexdump. > >>> > >>> [image: image.png] > >> > >> I'm afraid this is a text-only mailing list. Your screenshot has been > >> stripped out before any of us saw it. Could you copy and paste (DON'T > >> retype!) the error instead, so we can all read it? > >> > >> -- > >> Rhodri James *-* Kynesim Ltd > >> -- > >> https://mail.python.org/mailman/listinfo/python-list > >> > > > > > -- > Rhodri James *-* Kynesim Ltd > From rhodri at kynesim.co.uk Thu Dec 19 07:46:01 2019 From: rhodri at kynesim.co.uk (Rhodri James) Date: Thu, 19 Dec 2019 12:46:01 +0000 Subject: hexdump module installation error In-Reply-To: References: <303a8c7b-1c1b-12c0-bb0c-4cd70be8611a@kynesim.co.uk> <6ff313c1-d38b-5c5b-99cc-743413416f2f@kynesim.co.uk> Message-ID: <218a46b0-520b-aa2e-b02f-9621605b9a52@kynesim.co.uk> On 19/12/2019 12:43, tommy yama wrote: > Thanks for your quick response i did not expect. > I hope you see the error below in my response as i just copy and paste it. > > "no module named 'hexdump'." > > In addition, i tried to execute python3 hexdump.py. However, no such file > directory. > > from hexdump import hexdump > > ModuleNotFoundError: No module named 'hexdump' > > user at USERnoMacBook-Air LibraBrowser % python3 hexdump.py > > /usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS/Python: > can't open file 'hexdump.py': [Errno 2] No such file or directory > > user at USERnoMacBook-Air LibraBrowser % Huh. You've done a "pip3 install hexdump" so I don't know what might be happening here. Sorry. -- Rhodri James *-* Kynesim Ltd From lele at metapensiero.it Thu Dec 19 08:16:48 2019 From: lele at metapensiero.it (Lele Gaifax) Date: Thu, 19 Dec 2019 14:16:48 +0100 Subject: Cython, producing different modules from the same .pyx Message-ID: <87a77osben.fsf@metapensiero.it> Hi all, in my package, I would like to compile and distribute two different extension modules starting from the same .pyx file, just with different compilation flags and libraries. My first approach has been duplicating the Extension() entry in the setup.py(*), changing the first argument (that is, the name of the module). Although that did produce the alternative binary module, it could not be loaded because it contains the wrong PyInit_XXX(), where XXX is computed from the .pyx name, in my case "parser". I tried looking at Cython documentation, and even its sources, but couldn't find a way to alter that name. Did I miss something, or is the only way to duplicate the source .pyx file to a different name? Thanks in advance, ciao, lele. (*) https://github.com/lelit/pglast/blob/master/setup.py#L76 -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From rosuav at gmail.com Thu Dec 19 09:08:00 2019 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 20 Dec 2019 01:08:00 +1100 Subject: hexdump module installation error In-Reply-To: References: <303a8c7b-1c1b-12c0-bb0c-4cd70be8611a@kynesim.co.uk> <6ff313c1-d38b-5c5b-99cc-743413416f2f@kynesim.co.uk> Message-ID: On Thu, Dec 19, 2019 at 11:44 PM tommy yama wrote: > > Hi Rhodri, > > Thanks for your quick response i did not expect. > I hope you see the error below in my response as i just copy and paste it. > > "no module named 'hexdump'." > > In addition, i tried to execute python3 hexdump.py. However, no such file > directory. > > from hexdump import hexdump > Did you, at some point, have a file called hexdump.py that you were playing with? Sometimes, even after you delete a file with a conflicting name, its .pyc file hangs around. Blow away the __pycache__ directory to get rid of it. ChrisA From bluebox03 at gmail.com Thu Dec 19 09:09:12 2019 From: bluebox03 at gmail.com (tommy yama) Date: Thu, 19 Dec 2019 14:09:12 +0000 Subject: hexdump module installation error In-Reply-To: <218a46b0-520b-aa2e-b02f-9621605b9a52@kynesim.co.uk> References: <303a8c7b-1c1b-12c0-bb0c-4cd70be8611a@kynesim.co.uk> <6ff313c1-d38b-5c5b-99cc-743413416f2f@kynesim.co.uk> <218a46b0-520b-aa2e-b02f-9621605b9a52@kynesim.co.uk> Message-ID: Yes. thanks for your enthusiasm. i may raise this in the git On Thu, Dec 19, 2019 at 12:46 PM Rhodri James wrote: > On 19/12/2019 12:43, tommy yama wrote: > > Thanks for your quick response i did not expect. > > I hope you see the error below in my response as i just copy and paste > it. > > > > "no module named 'hexdump'." > > > > In addition, i tried to execute python3 hexdump.py. However, no such file > > directory. > > > > from hexdump import hexdump > > > > ModuleNotFoundError: No module named 'hexdump' > > > > user at USERnoMacBook-Air LibraBrowser % python3 hexdump.py > > > > > /usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS/Python: > > can't open file 'hexdump.py': [Errno 2] No such file or directory > > > > user at USERnoMacBook-Air LibraBrowser % > > Huh. You've done a "pip3 install hexdump" so I don't know what might be > happening here. Sorry. > > -- > Rhodri James *-* Kynesim Ltd > From onlinejudge95 at gmail.com Thu Dec 19 09:11:40 2019 From: onlinejudge95 at gmail.com (onlinejudge95) Date: Thu, 19 Dec 2019 19:41:40 +0530 Subject: Understanding of GIL Message-ID: Hi Devs, I am currently writing some custom Django commands for data updation, my workflow is like Fetch data from *PostgreSQL*. Call *Elasticsearch* for searching based on the data fetched from *PostgreSQL*. Query *PostgreSQL* and do an upsert behavior. I am using pandas data frame to hold my data during processing. The host we are using to run this jobs has a *CPython* interpreter as given by `platform.python_implementation()` I want to confirm whether *multithreading* would be a better choice here, given the fact that GIL is the biggest blocker(I agree it has to be there) for the same in CPython interpreters. In case further information is required do let me know. Thanks onlinejudge95 From pieter-l at vanoostrum.org Thu Dec 19 09:22:39 2019 From: pieter-l at vanoostrum.org (Pieter van Oostrum) Date: Thu, 19 Dec 2019 15:22:39 +0100 Subject: INHERITANCE in python3 References: <1681866660.1226419.1576728600632.ref@mail.yahoo.com> <1681866660.1226419.1576728600632@mail.yahoo.com> <250232d0-d5ce-4007-89a0-32e0fc23590a@www.fastmail.com> Message-ID: Random832 writes: > On Wed, Dec 18, 2019, at 23:10, vahid asadi via Python-list wrote: >> HI guys this is my first post on python mailing lists ever and i glad >> to do this. >> my problem here is why this attribute is not recognize by python and it >> raise an traceback error that said 'there is no such p.family >> attribute'. although i use multiple ??inheritance with 'super ' it not >> works. thanks for your help. >> >> ```?class Username:? ? def __init__(self,name,*args):? ? ? ? self.name >> = name >> class Userfamily:? ? def __init__(self,family,*args):? ? ? ? >> self.family = family >> class Person(Username,Userfamily):? ? def __init__(self,*args):? ? ? ? >> super().__init__(*args) >> >> >> p = Person("v","aaa")print(p.name)print(p.family)``` Please next time, supply a properly indented Python source, with only normal ASCII spaces, not no-break spaces, i.e. exactly like in your Python source code. > > The Username class also needs to call super(). In general, super() is > intended to be used with all classes that might be part of a multiple > inheritance hierarchy, not just the derived one. Just for safety, also add it to the Userfamily class. -- Pieter van Oostrum www: http://pieter.vanoostrum.org/ PGP key: [8DAE142BE17999C4] From pieter-l at vanoostrum.org Thu Dec 19 09:30:52 2019 From: pieter-l at vanoostrum.org (Pieter van Oostrum) Date: Thu, 19 Dec 2019 15:30:52 +0100 Subject: hexdump module installation error References: <303a8c7b-1c1b-12c0-bb0c-4cd70be8611a@kynesim.co.uk> <6ff313c1-d38b-5c5b-99cc-743413416f2f@kynesim.co.uk> Message-ID: tommy yama writes: > user at USERnoMacBook-Air LibraBrowser % python3 hexdump.py > > /usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS/Python: > can't open file 'hexdump.py': [Errno 2] No such file or directory > > user at USERnoMacBook-Air LibraBrowser % Could it be that your pip3 belongs to a different Python than the one above (for example a Python 3.8 or 3.6)? What is the output of 'pip3 --version' (without quotes)? -- Pieter van Oostrum www: http://pieter.vanoostrum.org/ PGP key: [8DAE142BE17999C4] From ethan at stoneleaf.us Thu Dec 19 12:35:49 2019 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 19 Dec 2019 09:35:49 -0800 Subject: Cython, producing different modules from the same .pyx In-Reply-To: <87a77osben.fsf@metapensiero.it> References: <87a77osben.fsf@metapensiero.it> Message-ID: <897ebc73-44f1-461f-28e1-9119c76538a4@stoneleaf.us> On 12/19/2019 05:16 AM, Lele Gaifax wrote: > in my package, I would like to compile and distribute two different extension > modules starting from the same .pyx file, just with different compilation > flags and libraries. If you don't get an answer here, you can try the Cython Users group: https://groups.google.com/forum/#!forum/cython-users -- ~Ethan~ From ethan at stoneleaf.us Thu Dec 19 12:28:21 2019 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 19 Dec 2019 09:28:21 -0800 Subject: INHERITANCE in python3 In-Reply-To: References: <1681866660.1226419.1576728600632.ref@mail.yahoo.com> <1681866660.1226419.1576728600632@mail.yahoo.com> <250232d0-d5ce-4007-89a0-32e0fc23590a@www.fastmail.com> Message-ID: <3e2e66f4-f5e4-0bf2-b9fc-93e71cbc3755@stoneleaf.us> On 12/19/2019 06:22 AM, Pieter van Oostrum wrote: > Random832 writes: >> On Wed, Dec 18, 2019, at 23:10, wrote: [vahid asadi] >>> my problem here is why this attribute is not recognize by python and it >>> raise an traceback error that said 'there is no such p.family >>> attribute'. although i use multiple inheritance with 'super ' it not >>> works. thanks for your help. >>> >>> class Username: >>> def __init__(self, name, *args): >>> self.name= name >>> >>> class Userfamily: >>> def __init__(self, family, *args): >>> self.family = family >>> >>> class Person(Username, Userfamily): >>> def __init__(self, *args): >>> super().__init__(*args) >>> >>> p = Person("v", "aaa") >>> print(p.name) >>> print(p.family) [Random32] >> The Username class also needs to call super(). In general, super() is >> intended to be used with all classes that might be part of a multiple >> inheritance hierarchy, not just the derived one. [Pieter van Oostrum] > Just for safety, also add it* to the Userfamily class. * a call to super() (in case anybody else misreads that like I did) -- ~Ethan~ From * at eli.users.panix.com Thu Dec 19 16:04:57 2019 From: * at eli.users.panix.com (Eli the Bearded) Date: Thu, 19 Dec 2019 21:04:57 +0000 (UTC) Subject: on sorting things References: Message-ID: In comp.lang.python, Peter Otten <__peter__ at web.de> wrote: > Eli the Bearded wrote: >> But what caught my eye most, as someone relatively new to Python but >> with long experience in C in Perl, is sorting doesn't take a s/C in /C and/ Ugh. >> *comparison* function, it takes a *key generator* function, and that >> function is supposed to transform the thing into something that the >> native comparison knows how to compare. >> >> This seems a strange choice, and I'm wondering if someone can explain >> the benefits of doing it that way to me. > > Python 2 started with a comparison function and then grew a key function. > With a key function you still have to compare items, you are just breaking > the comparison into two steps: [snip] Thanks for that good explanation. The benchmark comparison makes it very thorough. In my mind I gravitate towards the complicated sorts of sort that can be quickly compared for some sorts of keys and not as quickly for others. Consider a sort that first compares file size and if the same number of bytes, then compares file checksum. Any decently scaled real world implementation would memoize the checksum for speed, but only work it out for files that do not have a unique file size. The key method requires it worked out in advance for everything. But I see the key method handles the memoization under the hood for you, so those simpler, more common sorts of sort get an easy to see benefit. Elijah ------ even memoizing the stat() calls would help for large lists From rosuav at gmail.com Thu Dec 19 16:23:44 2019 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 20 Dec 2019 08:23:44 +1100 Subject: on sorting things In-Reply-To: References: Message-ID: On Fri, Dec 20, 2019 at 8:06 AM Eli the Bearded <*@eli.users.panix.com> wrote: > > In comp.lang.python, Peter Otten <__peter__ at web.de> wrote: > > Eli the Bearded wrote: > >> But what caught my eye most, as someone relatively new to Python but > >> with long experience in C in Perl, is sorting doesn't take a > > s/C in /C and/ > > Ugh. > > >> *comparison* function, it takes a *key generator* function, and that > >> function is supposed to transform the thing into something that the > >> native comparison knows how to compare. > >> > >> This seems a strange choice, and I'm wondering if someone can explain > >> the benefits of doing it that way to me. > > > > Python 2 started with a comparison function and then grew a key function. > > With a key function you still have to compare items, you are just breaking > > the comparison into two steps: > > [snip] > > Thanks for that good explanation. The benchmark comparison makes it > very thorough. > > In my mind I gravitate towards the complicated sorts of sort that can be > quickly compared for some sorts of keys and not as quickly for others. > > Consider a sort that first compares file size and if the same number of > bytes, then compares file checksum. Any decently scaled real world > implementation would memoize the checksum for speed, but only work it out > for files that do not have a unique file size. The key method requires > it worked out in advance for everything. > > But I see the key method handles the memoization under the hood for you, > so those simpler, more common sorts of sort get an easy to see benefit. > I guess that's a strange situation that might actually need this kind of optimization, but if you really do have that situation, you can make a magical key that behaves the way you want. class SizeChecksum: def __init__(self, fn): self.size = os.stat(fn).st_size self._checksum = None @property def checksum(self): if self._checksum is not None: return self._checksum ... def __lt__(self, other): if self.size != other.size: return self.size < other.size return self.checksum < other.checksum I can't remember exactly which comparison operators list.sort() uses, so you'd want to add another function here for an equality check, or maybe <=, or whichever is used. In any case, what matters is that the weird situations can still be handled, albeit by largely falling back on comparison semantics. But for the vast majority of situations, all you need is a key function that returns a number, a string, or a tuple of numbers and strings. ChrisA From pahome.chen at mirlab.org Thu Dec 19 23:33:25 2019 From: pahome.chen at mirlab.org (lampahome) Date: Fri, 20 Dec 2019 12:33:25 +0800 Subject: How to improve epoll speed when recv from kernel via netlink? Message-ID: I tried to receive msg from kernel via netlink of socket. And I use epoll to receive netlink events whenever it comes from kernel to user space. But I found the performance is poor e.g. epoll costs 90% time of execution time after I profile it by cProfile module. Are there any tips to improve this? From greg.ewing at canterbury.ac.nz Fri Dec 20 00:58:55 2019 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 20 Dec 2019 18:58:55 +1300 Subject: Cython, producing different modules from the same .pyx In-Reply-To: References: <87a77osben.fsf@metapensiero.it> Message-ID: On 20/12/19 2:16 am, Lele Gaifax wrote: > My first approach has been duplicating the Extension() entry in the > setup.py(*), changing the first argument (that is, the name of the module). > Although that did produce the alternative binary module, it could not be > loaded because it contains the wrong PyInit_XXX(), where XXX is computed from > the .pyx name, You could try creating a set of top-level .pyx stubs, each of which just 'include' the real code. -- Greg From p4j at j4d.net Fri Dec 20 07:12:24 2019 From: p4j at j4d.net (Pankaj Jangid) Date: Fri, 20 Dec 2019 17:42:24 +0530 Subject: [HN] How to Make Python Wait Message-ID: https://news.ycombinator.com/item?id=21834408 From rosuav at gmail.com Fri Dec 20 08:58:22 2019 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 21 Dec 2019 00:58:22 +1100 Subject: [HN] How to Make Python Wait In-Reply-To: References: Message-ID: On Fri, Dec 20, 2019 at 11:16 PM Pankaj Jangid wrote: > > https://news.ycombinator.com/item?id=21834408 > > Did you just post a blog article, then spam everywhere to try to get traffic, where your entire blog post is telling people worse ways to do a time.sleep()? ChrisA From arj.python at gmail.com Fri Dec 20 09:22:26 2019 From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer) Date: Fri, 20 Dec 2019 18:22:26 +0400 Subject: [HN] How to Make Python Wait In-Reply-To: References: Message-ID: Original from miguel grinberg From nt_mahmood at yahoo.com Fri Dec 20 02:22:00 2019 From: nt_mahmood at yahoo.com (Mahmood Naderan) Date: Fri, 20 Dec 2019 07:22:00 +0000 (UTC) Subject: Unable to install "collect" via pip3 References: <414413727.1095151.1576826520979.ref@mail.yahoo.com> Message-ID: <414413727.1095151.1576826520979@mail.yahoo.com> Hi I can install collect with pip for python2.7 $ pip install --user collect Collecting collect Using cached https://files.pythonhosted.org/packages/cf/5e/c0f0f51d081665374a2c219ea4ba23fb1e179b70dded96dc16606786d828/collect-0.1.1.tar.gz Collecting couchdbkit>=0.5.7 (from collect) Using cached https://files.pythonhosted.org/packages/a1/13/9e9ff695a385c44f62b4766341b97f2bd8b596962df2a0beabf358468b70/couchdbkit-0.6.5.tar.gz Collecting restkit>=4.2.2 (from couchdbkit>=0.5.7->collect) Downloading https://files.pythonhosted.org/packages/76/b9/d90120add1be718f853c53008cf5b62d74abad1d32bd1e7097dd913ae053/restkit-4.2.2.tar.gz (1.3MB) 100% |????????????????????????????????| 1.3MB 633kB/s Collecting http-parser>=0.8.3 (from restkit>=4.2.2->couchdbkit>=0.5.7->collect) Downloading https://files.pythonhosted.org/packages/07/c4/22e3c76c2313c26dd5f84f1205b916ff38ea951aab0c4544b6e2f5920d64/http-parser-0.8.3.tar.gz (83kB) 100% |????????????????????????????????| 92kB 2.4MB/s Collecting socketpool>=0.5.3 (from restkit>=4.2.2->couchdbkit>=0.5.7->collect) Downloading https://files.pythonhosted.org/packages/d1/39/fae99a735227234ffec389b252c6de2bc7816bf627f56b4c558dc46c85aa/socketpool-0.5.3.tar.gz Building wheels for collected packages: collect, couchdbkit, restkit, http-parser, socketpool Running setup.py bdist_wheel for collect ... done Stored in directory: /home/mnaderan/.cache/pip/wheels/b9/7c/7c/b09b334cc0e27b4f63ee9f6f19ca1f3db8672666a7e0f3d9cd Running setup.py bdist_wheel for couchdbkit ... done Stored in directory: /home/mnaderan/.cache/pip/wheels/f6/05/1b/f8f576ef18564bc68ab6e64f405e1263448036208cafb221e0 Running setup.py bdist_wheel for restkit ... done Stored in directory: /home/mnaderan/.cache/pip/wheels/48/c5/32/d0d25fb272791a68c49c26150f332d9b9492d0bc9ea0cdd2c7 Running setup.py bdist_wheel for http-parser ... done Stored in directory: /home/mnaderan/.cache/pip/wheels/22/db/06/cb609a3345e7aa87206de160f00cc6af364650d1139d904a25 Running setup.py bdist_wheel for socketpool ... done Stored in directory: /home/mnaderan/.cache/pip/wheels/93/f6/8c/65924848766618647078cb66b1d964e8b80876536e84517469 Successfully built collect couchdbkit restkit http-parser socketpool Installing collected packages: http-parser, socketpool, restkit, couchdbkit, collect Successfully installed collect-0.1.1 couchdbkit-0.6.5 http-parser-0.8.3 restkit-4.2.2 socketpool-0.5.3 However, pip3 fails with this error $ pip3 install --user collect Collecting collect Using cached https://files.pythonhosted.org/packages/cf/5e/c0f0f51d081665374a2c219ea4ba23fb1e179b70dded96dc16606786d828/collect-0.1.1.tar.gz Collecting couchdbkit>=0.5.7 (from collect) Using cached https://files.pythonhosted.org/packages/a1/13/9e9ff695a385c44f62b4766341b97f2bd8b596962df2a0beabf358468b70/couchdbkit-0.6.5.tar.gz Complete output from command python setup.py egg_info: Traceback (most recent call last): File "", line 1, in File "/tmp/pip-build-qf95n0tt/couchdbkit/setup.py", line 25, in long_description = file( NameError: name 'file' is not defined ---------------------------------------- Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-qf95n0tt/couchdbkit/ I can not figure out what is the problem. Any way to fix that? More info: $ which python /usr/bin/python $ ls -l /usr/bin/python lrwxrwxrwx 1 root root 9 Apr 16 2018 /usr/bin/python -> python2.7 $ which python3 /usr/bin/python3 $ ls -l /usr/bin/python3 lrwxrwxrwx 1 root root 9 Jun 21 2018 /usr/bin/python3 -> python3.6 Regards, Mahmood From rainer.woitok at gmail.com Fri Dec 20 07:19:26 2019 From: rainer.woitok at gmail.com (Dr Rainer Woitok) Date: Fri, 20 Dec 2019 13:19:26 +0100 Subject: Problems with "Tarfile.close()" Message-ID: <24060.48206.298944.886236@woitok.gmail.com> Greetings, One of my Python scripts basically does the following: source = tarfile.open(name=tar_archive , mode='r|*') dest = tarfile.open(fileobj=sys.stdout, mode='w|', format=fmt) . . . source.close() dest.close() In an attempt to move my Python scripts from Python 2.7 to Python 3.6 I ran into the problem that under Python 3.6 the call to "dest.close()" fails: Traceback (most recent call last): File ".../tar_archive.copy", line 137, in dest.close() File "/usr/lib64/python3.6/tarfile.py", line 1742, in close self.fileobj.close() File "/usr/lib64/python3.6/tarfile.py", line 467, in close self.fileobj.write(self.buf) TypeError: write() argument must be str, not bytes What am I doing wrong? By the way: since on some hosts this script is running on the transition from Python 2.7 to Python 3.x will not happen immediately, I need a solution which works with both versions. Sincerely, Rainer From rosuav at gmail.com Fri Dec 20 10:30:39 2019 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 21 Dec 2019 02:30:39 +1100 Subject: Unable to install "collect" via pip3 In-Reply-To: <414413727.1095151.1576826520979@mail.yahoo.com> References: <414413727.1095151.1576826520979.ref@mail.yahoo.com> <414413727.1095151.1576826520979@mail.yahoo.com> Message-ID: On Sat, Dec 21, 2019 at 2:25 AM Mahmood Naderan via Python-list wrote: > > Hi > > I can install collect with pip for python2.7 > $ pip install --user collect > However, pip3 fails with this error > $ pip3 install --user collect > NameError: name 'file' is not defined > > I can not figure out what is the problem. Any way to fix that? > Are you trying to install this package? https://pypi.org/project/collect/ It's alpha software that does not claim to support Python 3. The last release was in 2011. The web site it links to is defunct. Maybe that wasn't what you intended to install? Check the spelling of the package you're trying to install - maybe it's named slightly differently. ChrisA From rosuav at gmail.com Fri Dec 20 10:34:28 2019 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 21 Dec 2019 02:34:28 +1100 Subject: Problems with "Tarfile.close()" In-Reply-To: <24060.48206.298944.886236@woitok.gmail.com> References: <24060.48206.298944.886236@woitok.gmail.com> Message-ID: On Sat, Dec 21, 2019 at 2:29 AM Dr Rainer Woitok wrote: > > Greetings, > > One of my Python scripts basically does the following: > > source = tarfile.open(name=tar_archive , mode='r|*') > dest = tarfile.open(fileobj=sys.stdout, mode='w|', format=fmt) > > . > . > . > > source.close() > dest.close() > > In an attempt to move my Python scripts from Python 2.7 to Python 3.6 I > ran into the problem that under Python 3.6 the call to "dest.close()" > fails: (I think the fact that it fails when you close the file is a red herring; it would fail at some point, and it happens to hold things over until it closes.) > Traceback (most recent call last): > File ".../tar_archive.copy", line 137, in > dest.close() > File "/usr/lib64/python3.6/tarfile.py", line 1742, in close > self.fileobj.close() > File "/usr/lib64/python3.6/tarfile.py", line 467, in close > self.fileobj.write(self.buf) > TypeError: write() argument must be str, not bytes > > What am I doing wrong? By the way: since on some hosts this script is > running on the transition from Python 2.7 to Python 3.x will not happen > immediately, I need a solution which works with both versions. > Possibly the easiest fix would be to open all files in binary mode. I think your content is binary anyway by the look of it. Just add the letter "b" to the end of your open file modes, and on Py2, it'll ensure that no newline conversion happens (mainly an issue on Windows, which is why you probably don't feel the need to do this), but on Py3, it means that it expects bytestrings everywhere. If that *doesn't* work, then you may need to mark some of your strings as binary, if you have unadorned strings containing ASCII data. ChrisA From ethan at stoneleaf.us Fri Dec 20 10:41:51 2019 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 20 Dec 2019 07:41:51 -0800 Subject: Problems with "Tarfile.close()" In-Reply-To: <24060.48206.298944.886236@woitok.gmail.com> References: <24060.48206.298944.886236@woitok.gmail.com> Message-ID: <213f5084-5007-8a87-5ac3-83cbd734e585@stoneleaf.us> On 12/20/2019 04:19 AM, Dr Rainer Woitok wrote: > One of my Python scripts basically does the following: > > source = tarfile.open(name=tar_archive , mode='r|*') > dest = tarfile.open(fileobj=sys.stdout, mode='w|', format=fmt) > > . > . > . > > source.close() > dest.close() > > In an attempt to move my Python scripts from Python 2.7 to Python 3.6 I > ran into the problem that under Python 3.6 the call to "dest.close()" > fails: > > Traceback (most recent call last): > File ".../tar_archive.copy", line 137, in > dest.close() > File "/usr/lib64/python3.6/tarfile.py", line 1742, in close > self.fileobj.close() > File "/usr/lib64/python3.6/tarfile.py", line 467, in close > self.fileobj.write(self.buf) > TypeError: write() argument must be str, not bytes In Python 3 `sys.stdout` is a character interface, not bytes. There are a couple solutions to the Python 3 aspect of the problem here: https://stackoverflow.com/q/908331/208880 If those answers do not work on Python 2 you'll need to detect which version you are on and act appropriately, perhaps hiding that bit of complexity in a function or class. -- ~Ethan~ From barry at barrys-emacs.org Fri Dec 20 10:01:41 2019 From: barry at barrys-emacs.org (Barry Scott) Date: Fri, 20 Dec 2019 15:01:41 +0000 Subject: How to improve epoll speed when recv from kernel via netlink? In-Reply-To: References: Message-ID: > On 20 Dec 2019, at 04:33, lampahome wrote: > > I tried to receive msg from kernel via netlink of socket. > > And I use epoll to receive netlink events whenever it comes from kernel to > user space. > > But I found the performance is poor e.g. epoll costs 90% time of execution > time after I profile it by cProfile module. > > Are there any tips to improve this? cProfile is telling you how long the code is waiting for epoll to return. It is not telling that epoll is a problem. Barry > -- > https://mail.python.org/mailman/listinfo/python-list > From p4j at j4d.net Fri Dec 20 12:16:38 2019 From: p4j at j4d.net (p4j at j4d.net) Date: Fri, 20 Dec 2019 22:46:38 +0530 Subject: [HN] How to Make Python Wait References: Message-ID: >> https://news.ycombinator.com/item?id=21834408 > Did you just post a blog article, then spam everywhere to try to get > traffic, where your entire blog post is telling people worse ways to > do a time.sleep()? Blog post is not mine. I have a habit of posting good things to HN. When I got notification that it is getting good traction, people are discussing, then I posted the discussion link here. If that is not liked here then I won't post again. From rainer.woitok at gmail.com Fri Dec 20 12:20:11 2019 From: rainer.woitok at gmail.com (Dr Rainer Woitok) Date: Fri, 20 Dec 2019 18:20:11 +0100 Subject: Problems with "Tarfile.close()" In-Reply-To: Msg <213f5084-5007-8a87-5ac3-83cbd734e585@stoneleaf.us> of 2019-12-20 07:41:51 -0800 from ethan@stoneleaf.us References: <24060.48206.298944.886236@woitok.gmail.com> <213f5084-5007-8a87-5ac3-83cbd734e585@stoneleaf.us> Message-ID: <24061.715.861556.185941@woitok.gmail.com> Ethan, On Friday, 2019-12-20 07:41:51 -0800, you wrote: > ... > In Python 3 `sys.stdout` is a character interface, not bytes. Does that mean that with Python 3 "Tarfile" is no longer able to write the "tar" file to a pipe? Or is there now another way to write to a pipe? And if that new way also worked with Python 2, it would be even better ... :-) > There are a couple solutions to the Python 3 aspect of the problem here: > > https://stackoverflow.com/q/908331/208880 Using "sys.stdout.buffer" seems to work in Python 3 (at least with my current rather trivial test case) but does not work in Python 2. Quest- ion: what is the cheapest way to retrieve the Python version the script is executing in? Sincerely, Rainer From __peter__ at web.de Fri Dec 20 13:01:34 2019 From: __peter__ at web.de (Peter Otten) Date: Fri, 20 Dec 2019 19:01:34 +0100 Subject: on sorting things References: Message-ID: Eli the Bearded wrote: > In comp.lang.python, Peter Otten <__peter__ at web.de> wrote: >> Eli the Bearded wrote: >>> But what caught my eye most, as someone relatively new to Python but >>> with long experience in C in Perl, is sorting doesn't take a > > s/C in /C and/ > > Ugh. > >>> *comparison* function, it takes a *key generator* function, and that >>> function is supposed to transform the thing into something that the >>> native comparison knows how to compare. >>> >>> This seems a strange choice, and I'm wondering if someone can explain >>> the benefits of doing it that way to me. >> >> Python 2 started with a comparison function and then grew a key function. >> With a key function you still have to compare items, you are just >> breaking the comparison into two steps: > > [snip] > > Thanks for that good explanation. The benchmark comparison makes it > very thorough. > > In my mind I gravitate towards the complicated sorts of sort that can be > quickly compared for some sorts of keys and not as quickly for others. > > Consider a sort that first compares file size and if the same number of > bytes, then compares file checksum. Any decently scaled real world > implementation would memoize the checksum for speed, but only work it out > for files that do not have a unique file size. The key method requires > it worked out in advance for everything. Oscar already mentioned the functools.cmp_to_key decorator which makes this a non-issue: def mycmp(a, b): ... files.sort(key=cmp_to_key(mycmp)) Applied to your example, with memoization: # untested def cmp(a, b): return (a > b)-(a < b) def make_file_key(): size = functools.lru_cache(None)(getsize) checksum = functools.lru_cache(None)(getchecksum) @functools.cmp_to_key def file_key(a, b): return cmp(size(a), size(b)) or cmp(checksum(a), checksum(b)) return file_key files.sort(key=make_file_key()) > But I see the key method handles the memoization under the hood for you, > so those simpler, more common sorts of sort get an easy to see benefit. > > Elijah > ------ > even memoizing the stat() calls would help for large lists PS: If you are sorting files by size and checksum as part of a deduplication effort consider using dict-s instead: def grouped(items, key): result = defaultdict(list) for item in items: result[key(item)].append(item) return result for same_size in grouped(files, key=getsize).values(): if len(same_size) > 1: for same_checksum in grouped(same_size, key=getchecksum).values(): if len(same_checksum) > 1: print(same_checksum) From rosuav at gmail.com Fri Dec 20 13:11:21 2019 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 21 Dec 2019 05:11:21 +1100 Subject: on sorting things In-Reply-To: References: Message-ID: On Sat, Dec 21, 2019 at 5:03 AM Peter Otten <__peter__ at web.de> wrote: > PS: If you are sorting files by size and checksum as part of a deduplication > effort consider using dict-s instead: Yeah, I'd agree if that's the purpose. But let's say the point is to have a guaranteed-stable ordering of files that are primarily to be sorted by file size - in order to ensure that two files are in the same order every time you refresh the view, they get sorted by their checksums. There ARE good reasons to do weird things with sorting, and a custom key object (either with cmp_to_key or directly implemented) can do that. ChrisA From lele at metapensiero.it Fri Dec 20 13:33:47 2019 From: lele at metapensiero.it (Lele Gaifax) Date: Fri, 20 Dec 2019 19:33:47 +0100 Subject: Cython, producing different modules from the same .pyx References: <87a77osben.fsf@metapensiero.it> Message-ID: <87h81uyhh0.fsf@metapensiero.it> Greg Ewing writes: > You could try creating a set of top-level .pyx stubs, each of > which just 'include' the real code. Thank you, will try this approach! ciao, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From lele at metapensiero.it Fri Dec 20 13:35:04 2019 From: lele at metapensiero.it (Lele Gaifax) Date: Fri, 20 Dec 2019 19:35:04 +0100 Subject: Cython, producing different modules from the same .pyx References: <87a77osben.fsf@metapensiero.it> <897ebc73-44f1-461f-28e1-9119c76538a4@stoneleaf.us> Message-ID: <87a77myhev.fsf@metapensiero.it> Ethan Furman writes: > If you don't get an answer here, you can try the Cython Users group: Thanks, reposted the same question there. ciao, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From __peter__ at web.de Fri Dec 20 13:59:53 2019 From: __peter__ at web.de (Peter Otten) Date: Fri, 20 Dec 2019 19:59:53 +0100 Subject: on sorting things References: Message-ID: Chris Angelico wrote: > On Sat, Dec 21, 2019 at 5:03 AM Peter Otten <__peter__ at web.de> wrote: >> PS: If you are sorting files by size and checksum as part of a >> deduplication effort consider using dict-s instead: > > Yeah, I'd agree if that's the purpose. But let's say the point is to > have a guaranteed-stable ordering of files that are primarily to be > sorted by file size - in order to ensure that two files are in the > same order every time you refresh the view, they get sorted by their > checksums. One thing that struck me about Eli's example is that it features two key functions rather than a complex comparison. If sort() would accept a sequence of key functions each function could be used to sort slices that compare equal when using the previous key. > There ARE good reasons to do weird things with sorting, and a custom > key object (either with cmp_to_key or directly implemented) can do > that. Indeed. From __peter__ at web.de Fri Dec 20 14:12:11 2019 From: __peter__ at web.de (Peter Otten) Date: Fri, 20 Dec 2019 20:12:11 +0100 Subject: Problems with "Tarfile.close()" References: <24060.48206.298944.886236@woitok.gmail.com> <213f5084-5007-8a87-5ac3-83cbd734e585@stoneleaf.us> <24061.715.861556.185941@woitok.gmail.com> Message-ID: Dr Rainer Woitok wrote: > Ethan, > > On Friday, 2019-12-20 07:41:51 -0800, you wrote: > >> ... >> In Python 3 `sys.stdout` is a character interface, not bytes. > > Does that mean that with Python 3 "Tarfile" is no longer able to write > the "tar" file to a pipe? Or is there now another way to write to a > pipe? And if that new way also worked with Python 2, it would be even > better ... :-) > >> There are a couple solutions to the Python 3 aspect of the problem here: >> >> https://stackoverflow.com/q/908331/208880 > > Using "sys.stdout.buffer" seems to work in Python 3 (at least with my > current rather trivial test case) but does not work in Python 2. Quest- > ion: what is the cheapest way to retrieve the Python version the script > is executing in? While I didn't look into the stackoverflow page an easy way to get something that accepts bytes may be # untested stdout = sys.stdout try: stdout = stdout.buffer except AttributeError: pass tarfile.open(fileobj=stdout, ...) From rosuav at gmail.com Fri Dec 20 14:44:03 2019 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 21 Dec 2019 06:44:03 +1100 Subject: on sorting things In-Reply-To: References: Message-ID: On Sat, Dec 21, 2019 at 6:01 AM Peter Otten <__peter__ at web.de> wrote: > > Chris Angelico wrote: > > > On Sat, Dec 21, 2019 at 5:03 AM Peter Otten <__peter__ at web.de> wrote: > >> PS: If you are sorting files by size and checksum as part of a > >> deduplication effort consider using dict-s instead: > > > > Yeah, I'd agree if that's the purpose. But let's say the point is to > > have a guaranteed-stable ordering of files that are primarily to be > > sorted by file size - in order to ensure that two files are in the > > same order every time you refresh the view, they get sorted by their > > checksums. > > One thing that struck me about Eli's example is that it features two key > functions rather than a complex comparison. > > If sort() would accept a sequence of key functions each function could be > used to sort slices that compare equal when using the previous key. > That would basically make it a comparison function, not a key function :) The value of the key function is that it's called exactly once per element, and the result is what you sort by. There's no correlation to any other element. It's effectively this: # sortme.sort(key=keyfunc) keys = sortme.map(keyfunc) keys.sort(keep_in_parallel=sortme) Which is why cmp_to_key is the correct solution to that problem. (For cases where you don't mind pre-calling both key functions, of course, it's equivalent to a single key function that returns a tuple.) ChrisA From rosuav at gmail.com Fri Dec 20 14:45:24 2019 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 21 Dec 2019 06:45:24 +1100 Subject: Problems with "Tarfile.close()" In-Reply-To: References: <24060.48206.298944.886236@woitok.gmail.com> <213f5084-5007-8a87-5ac3-83cbd734e585@stoneleaf.us> <24061.715.861556.185941@woitok.gmail.com> Message-ID: On Sat, Dec 21, 2019 at 6:13 AM Peter Otten <__peter__ at web.de> wrote: > > Dr Rainer Woitok wrote: > > > Ethan, > > > > On Friday, 2019-12-20 07:41:51 -0800, you wrote: > > > >> ... > >> In Python 3 `sys.stdout` is a character interface, not bytes. > > > > Does that mean that with Python 3 "Tarfile" is no longer able to write > > the "tar" file to a pipe? Or is there now another way to write to a > > pipe? And if that new way also worked with Python 2, it would be even > > better ... :-) > > > >> There are a couple solutions to the Python 3 aspect of the problem here: > >> > >> https://stackoverflow.com/q/908331/208880 > > > > Using "sys.stdout.buffer" seems to work in Python 3 (at least with my > > current rather trivial test case) but does not work in Python 2. Quest- > > ion: what is the cheapest way to retrieve the Python version the script > > is executing in? > > While I didn't look into the stackoverflow page an easy way to get something > that accepts bytes may be > > # untested > stdout = sys.stdout > try: > stdout = stdout.buffer > except AttributeError: > pass This construct can be simplified down to: stdout = getattr(sys.stdout, "buffer", sys.stdout) ChrisA From tjreedy at udel.edu Fri Dec 20 14:54:49 2019 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 20 Dec 2019 14:54:49 -0500 Subject: Unable to install "collect" via pip3 In-Reply-To: <414413727.1095151.1576826520979@mail.yahoo.com> References: <414413727.1095151.1576826520979.ref@mail.yahoo.com> <414413727.1095151.1576826520979@mail.yahoo.com> Message-ID: On 12/20/2019 2:22 AM, Mahmood Naderan via Python-list wrote: > However, pip3 fails with this error > $ pip3 install --user collect > Collecting collect > Using cached https://files.pythonhosted.org/packages/cf/5e/c0f0f51d081665374a2c219ea4ba23fb1e179b70dded96dc16606786d828/collect-0.1.1.tar.gz > Collecting couchdbkit>=0.5.7 (from collect) > Using cached https://files.pythonhosted.org/packages/a1/13/9e9ff695a385c44f62b4766341b97f2bd8b596962df2a0beabf358468b70/couchdbkit-0.6.5.tar.gz > Complete output from command python setup.py egg_info: > Traceback (most recent call last): > File "", line 1, in > File "/tmp/pip-build-qf95n0tt/couchdbkit/setup.py", line 25, in > long_description = file( > NameError: name 'file' is not defined The builtin function 'file' does not exist in 3.x. pip3 is trying to install 2.x code. I suspect this is because the package is not properly labelled as 2.x only. (Some 2.x code will run unaltered on 3.x, so pip should try even if 3.x is not specified.) -- Terry Jan Reedy From barry at barrys-emacs.org Fri Dec 20 17:10:24 2019 From: barry at barrys-emacs.org (Barry) Date: Fri, 20 Dec 2019 22:10:24 +0000 Subject: Unable to install "collect" via pip3 In-Reply-To: <414413727.1095151.1576826520979@mail.yahoo.com> References: <414413727.1095151.1576826520979@mail.yahoo.com> Message-ID: > On 20 Dec 2019, at 15:27, Mahmood Naderan via Python-list wrote: > > ?Hi > > I can install collect with pip for python2.7 > $ pip install --user collect > Collecting collect > Using cached https://files.pythonhosted.org/packages/cf/5e/c0f0f51d081665374a2c219ea4ba23fb1e179b70dded96dc16606786d828/collect-0.1.1.tar.gz > Collecting couchdbkit>=0.5.7 (from collect) > Using cached https://files.pythonhosted.org/packages/a1/13/9e9ff695a385c44f62b4766341b97f2bd8b596962df2a0beabf358468b70/couchdbkit-0.6.5.tar.gz > Collecting restkit>=4.2.2 (from couchdbkit>=0.5.7->collect) > Downloading https://files.pythonhosted.org/packages/76/b9/d90120add1be718f853c53008cf5b62d74abad1d32bd1e7097dd913ae053/restkit-4.2.2.tar.gz (1.3MB) > 100% |????????????????????????????????| 1.3MB 633kB/s > Collecting http-parser>=0.8.3 (from restkit>=4.2.2->couchdbkit>=0.5.7->collect) > Downloading https://files.pythonhosted.org/packages/07/c4/22e3c76c2313c26dd5f84f1205b916ff38ea951aab0c4544b6e2f5920d64/http-parser-0.8.3.tar.gz (83kB) > 100% |????????????????????????????????| 92kB 2.4MB/s > Collecting socketpool>=0.5.3 (from restkit>=4.2.2->couchdbkit>=0.5.7->collect) > Downloading https://files.pythonhosted.org/packages/d1/39/fae99a735227234ffec389b252c6de2bc7816bf627f56b4c558dc46c85aa/socketpool-0.5.3.tar.gz > Building wheels for collected packages: collect, couchdbkit, restkit, http-parser, socketpool > Running setup.py bdist_wheel for collect ... done > Stored in directory: /home/mnaderan/.cache/pip/wheels/b9/7c/7c/b09b334cc0e27b4f63ee9f6f19ca1f3db8672666a7e0f3d9cd > Running setup.py bdist_wheel for couchdbkit ... done > Stored in directory: /home/mnaderan/.cache/pip/wheels/f6/05/1b/f8f576ef18564bc68ab6e64f405e1263448036208cafb221e0 > Running setup.py bdist_wheel for restkit ... done > Stored in directory: /home/mnaderan/.cache/pip/wheels/48/c5/32/d0d25fb272791a68c49c26150f332d9b9492d0bc9ea0cdd2c7 > Running setup.py bdist_wheel for http-parser ... done > Stored in directory: /home/mnaderan/.cache/pip/wheels/22/db/06/cb609a3345e7aa87206de160f00cc6af364650d1139d904a25 > Running setup.py bdist_wheel for socketpool ... done > Stored in directory: /home/mnaderan/.cache/pip/wheels/93/f6/8c/65924848766618647078cb66b1d964e8b80876536e84517469 > Successfully built collect couchdbkit restkit http-parser socketpool > Installing collected packages: http-parser, socketpool, restkit, couchdbkit, collect > Successfully installed collect-0.1.1 couchdbkit-0.6.5 http-parser-0.8.3 restkit-4.2.2 socketpool-0.5.3 > However, pip3 fails with this error > $ pip3 install --user collect > Collecting collect > Using cached https://files.pythonhosted.org/packages/cf/5e/c0f0f51d081665374a2c219ea4ba23fb1e179b70dded96dc16606786d828/collect-0.1.1.tar.gz > Collecting couchdbkit>=0.5.7 (from collect) > Using cached https://files.pythonhosted.org/packages/a1/13/9e9ff695a385c44f62b4766341b97f2bd8b596962df2a0beabf358468b70/couchdbkit-0.6.5.tar.gz > Complete output from command python setup.py egg_info: > Traceback (most recent call last): > File "", line 1, in > File "/tmp/pip-build-qf95n0tt/couchdbkit/setup.py", line 25, in > long_description = file( > NameError: name 'file' is not defined My guess is that file is python 2 only. Couchdbkit needs porting to python 3. Barry > > ---------------------------------------- > Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-qf95n0tt/couchdbkit/ > I can not figure out what is the problem. Any way to fix that? > > More info: > $ which python > /usr/bin/python > $ ls -l /usr/bin/python > lrwxrwxrwx 1 root root 9 Apr 16 2018 /usr/bin/python -> python2.7 > $ which python3 > /usr/bin/python3 > $ ls -l /usr/bin/python3 > lrwxrwxrwx 1 root root 9 Jun 21 2018 /usr/bin/python3 -> python3.6 > > > > Regards, > Mahmood > -- > https://mail.python.org/mailman/listinfo/python-list From greg.ewing at canterbury.ac.nz Fri Dec 20 20:50:45 2019 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 21 Dec 2019 14:50:45 +1300 Subject: How to extend an object? In-Reply-To: References: Message-ID: On 21/12/19 1:59 am, Stefan Ram wrote: > I would like to add a method to a string. > > This is not possible in Python? It's not possible. Built-in classes can't have methods added to them. You can define your own subclass of str and give it whatever methods you want. But in your case: > for s in 'abc'.like( '(.)' ): I would recommend making it a stand-alone function instead, so that you would write for s in like('abc', '(.)'): -- Greg From wanghao7915 at gmail.com Sat Dec 21 00:04:20 2019 From: wanghao7915 at gmail.com (wanghao7915 at gmail.com) Date: Fri, 20 Dec 2019 21:04:20 -0800 (PST) Subject: How to combine a group of lists together Message-ID: <4bd40fe1-b703-491a-a573-cfb4d44403fb@googlegroups.com> I utilized .get_text() + .split() functions and obtain this: spe= ['$278.86as', 'of', 'Dec', '20,', '2019,', '06:47', 'PST', '-', 'Details'] ['4.7', 'inches'] ['750', 'x', '1334'] ['64', 'GB'] ['2', 'GB'] ['Apple', 'A11', 'Bionic,', 'Hexa-Core,', '(2x', 'Monsoon', '+', '4x', 'Mistral)'] ['5.22', 'ounces'] ['No'] ['No'] ['12', 'MP'] ['7', 'MP'] ['1821', 'mah'] ['No'] ['Yes,', 'on', 'the', 'front'] ['None'] ['iOS', '(Last', 'OS', 'info:', 'IOS', '11)'] ['September', '2017'] They have a shared name 'spe', but no concrete names for each list. If I use spe[0], I can only get all the first element of each list. How can I mix all of these elements into a single list like: ['$278.86as', 'of', 'Dec', '20,', '2019,', '06:47', 'PST', '-', 'Details','4.7', 'inches'......] Thanks! From cs at cskk.id.au Sat Dec 21 00:31:36 2019 From: cs at cskk.id.au (Cameron Simpson) Date: Sat, 21 Dec 2019 16:31:36 +1100 Subject: How to combine a group of lists together In-Reply-To: <4bd40fe1-b703-491a-a573-cfb4d44403fb@googlegroups.com> References: <4bd40fe1-b703-491a-a573-cfb4d44403fb@googlegroups.com> Message-ID: <20191221053136.GA95909@cskk.homeip.net> On 20Dec2019 21:04, wanghao7915 at gmail.com wrote: >I utilized .get_text() + .split() functions and obtain this: >spe= >['$278.86as', 'of', 'Dec', '20,', '2019,', '06:47', 'PST', '-', 'Details'] >['4.7', 'inches'] >['750', 'x', '1334'] >['64', 'GB'] >['2', 'GB'] >['Apple', 'A11', 'Bionic,', 'Hexa-Core,', '(2x', 'Monsoon', '+', '4x', 'Mistral)'] >['5.22', 'ounces'] >['No'] >['No'] >['12', 'MP'] >['7', 'MP'] >['1821', 'mah'] >['No'] >['Yes,', 'on', 'the', 'front'] >['None'] >['iOS', '(Last', 'OS', 'info:', 'IOS', '11)'] >['September', '2017'] > >They have a shared name 'spe', but no concrete names for each list. Does that mean you have a loop, with spe being computed in each loop pass? It would help to see your code. >If I use spe[0], I can only get all the first element of each list. Indeed, since that designates the first element. >How can I mix all of these elements into a single list like: >['$278.86as', 'of', 'Dec', '20,', '2019,', '06:47', 'PST', '-', >'Details','4.7', 'inches'......] Have a look at the chain() function from the itertools module. Or at the extend() method of the list class. Either will help you compose a longer list from a collection of individual lists. If you're still stuck, post your code in a followup message and ask sepecific questions about your attempts to build a bigger list from various smaller ones. Cheers, Cameron Simpson From PythonList at DancesWithMice.info Sat Dec 21 00:48:43 2019 From: PythonList at DancesWithMice.info (DL Neil) Date: Sat, 21 Dec 2019 18:48:43 +1300 Subject: How to combine a group of lists together In-Reply-To: <4bd40fe1-b703-491a-a573-cfb4d44403fb@googlegroups.com> References: <4bd40fe1-b703-491a-a573-cfb4d44403fb@googlegroups.com> Message-ID: <636aa1d7-7fd4-b59d-f511-8d7084669022@DancesWithMice.info> On 21/12/19 6:04 pm, wanghao7915 at gmail.com wrote: > I utilized .get_text() + .split() functions and obtain this: > spe= > ['$278.86as', 'of', 'Dec', '20,', '2019,', '06:47', 'PST', '-', 'Details'] > ['4.7', 'inches'] > ['750', 'x', '1334'] > ['64', 'GB'] > ['2', 'GB'] > ['Apple', 'A11', 'Bionic,', 'Hexa-Core,', '(2x', 'Monsoon', '+', '4x', 'Mistral)'] > ['5.22', 'ounces'] > ['No'] > ['No'] > ['12', 'MP'] > ['7', 'MP'] > ['1821', 'mah'] > ['No'] > ['Yes,', 'on', 'the', 'front'] > ['None'] > ['iOS', '(Last', 'OS', 'info:', 'IOS', '11)'] > ['September', '2017'] > > They have a shared name 'spe', but no concrete names for each list. > If I use spe[0], I can only get all the first element of each list. > > How can I mix all of these elements into a single list like: > ['$278.86as', 'of', 'Dec', '20,', '2019,', '06:47', 'PST', '-', 'Details','4.7', 'inches'......] What is type( spe )? Please copy-paste code/session into your email msg - the above is NOT Python. Your best solution may be to use a loop (instead of the constant 0, as list-argument). The answer to your specific question is 'in the manual': list.extend() (but a loop will likely still be required...) WebRef: https://docs.python.org/3/library/stdtypes.html#sequence-types-list-tuple-range -- Regards =dn From PythonList at DancesWithMice.info Sat Dec 21 01:00:04 2019 From: PythonList at DancesWithMice.info (DL Neil) Date: Sat, 21 Dec 2019 19:00:04 +1300 Subject: How to extend an object? In-Reply-To: References: Message-ID: <9bb4965d-47f7-dabc-93a4-62f673faf62c@DancesWithMice.info> On 21/12/19 2:50 pm, Greg Ewing wrote: > On 21/12/19 1:59 am, Stefan Ram wrote: >> ?? I would like to add a method to a string. >> >> ?? This is not possible in Python? > > It's not possible. Built-in classes can't have methods added > to them. > > You can define your own subclass of str and give it whatever > methods you want. > > But in your case: > >> for s in 'abc'.like( '(.)' ): > > I would recommend making it a stand-alone function instead, > so that you would write > > ? for s in like('abc', '(.)'): Contrarily: sub-classing str and making like() a method strikes me as the way to go, particularly if the 'abc' disappears into the instance's __init__() - and better still if the '(.)' may also be absorbed as a class/instance attribute, and even more-so if like() is not the only method/stand-alone function required for this category of strings... but... choose a self-documenting name for the class, and choose a better-documenting name for the method. instance = class( "abc" ) ... for s in instance.method(): is clean, self-documenting (or will be once you've done your thing), easier to test (fewer args => easier assured testing). -- Regards =dn From wanghao7915 at gmail.com Sat Dec 21 01:47:31 2019 From: wanghao7915 at gmail.com (wanghao7915 at gmail.com) Date: Fri, 20 Dec 2019 22:47:31 -0800 (PST) Subject: How to combine a group of lists together In-Reply-To: <4bd40fe1-b703-491a-a573-cfb4d44403fb@googlegroups.com> References: <4bd40fe1-b703-491a-a573-cfb4d44403fb@googlegroups.com> Message-ID: <1497d8b4-3875-436d-b0da-90e4448197e7@googlegroups.com> thanks for responding! Now I've solved this problem! My former code is: for j in data: spe = j.get_text() l_spe = spe.split() print(l_spe) Then I get the result: ['$278.86as', 'of', 'Dec', '20,', '2019,', '06:47', 'PST', '-', 'Details'] ['4.7', 'inches'] ['750', 'x', '1334'] ['64', 'GB'] ['2', 'GB'] ['Apple', 'A11', 'Bionic,', 'Hexa-Core,', '(2x', 'Monsoon', '+', '4x', 'Mistral)'] ['5.22', 'ounces'] ['No'] ['No'] ['12', 'MP'] ['7', 'MP'] ['1821', 'mah'] ['No'] ['Yes,', 'on', 'the', 'front'] ['None'] ['iOS', '(Last', 'OS', 'info:', 'IOS', '11)'] ['September', '2017'] Now my code is: for j in data: spe = j.get_text() temp_li.extend(spe.split()) print(temp_li) Then: ['$278.86as', 'of', 'Dec', '20,', '2019,', '06:47', 'PST', '-', 'Details', '4.7', 'inches', '750', 'x', '1334', '64', 'GB', '2', 'GB', 'Apple', 'A11', 'Bionic,', 'Hexa-Core,', '(2x', 'Monsoon', '+', '4x', 'Mistral)', '5.22', 'ounces', 'No', 'No', '12', 'MP', '7', 'MP', '1821', 'mah', 'No', 'Yes,', 'on', 'the', 'front', 'None', 'iOS', '(Last', 'OS', 'info:', 'IOS', '11)', 'September', '2017'] Bravo! From wanghao7915 at gmail.com Sat Dec 21 01:52:08 2019 From: wanghao7915 at gmail.com (wanghao7915 at gmail.com) Date: Fri, 20 Dec 2019 22:52:08 -0800 (PST) Subject: How to combine a group of lists together In-Reply-To: References: <4bd40fe1-b703-491a-a573-cfb4d44403fb@googlegroups.com> <20191221053136.GA95909@cskk.homeip.net> Message-ID: <700e2b65-67b7-4e3e-b5bf-4a2b033862ff@googlegroups.com> Thanks! Actually, I neglected that my "print" is in the "for", so I get a group of lists. Now I put the "print" out of the iteration then use .extend to append each list I produce. Finally, print and get my ideal result. Haha, I'm too naive, so is my English. From rainer.woitok at gmail.com Sat Dec 21 03:15:54 2019 From: rainer.woitok at gmail.com (Dr Rainer Woitok) Date: Sat, 21 Dec 2019 09:15:54 +0100 Subject: Problems with "Tarfile.close()" In-Reply-To: Msg of 2019-12-21 06:45:24 +1100 from rosuav@gmail.com References: <24060.48206.298944.886236@woitok.gmail.com> <213f5084-5007-8a87-5ac3-83cbd734e585@stoneleaf.us> <24061.715.861556.185941@woitok.gmail.com> Message-ID: <24061.54458.482156.152148@woitok.gmail.com> Chris, On Saturday, 2019-12-21 06:45:24 +1100, you wrote: > ... > This construct can be simplified down to: > > stdout = getattr(sys.stdout, "buffer", sys.stdout) Nice! And with my admittedly trivial test case it correctly works under both, Python 2 and Python 3. Now I'll do some more serious testing ... Sincerely, Rainer From bluebox03 at gmail.com Sat Dec 21 03:44:51 2019 From: bluebox03 at gmail.com (tommy yama) Date: Sat, 21 Dec 2019 08:44:51 +0000 Subject: hexdump module installation error In-Reply-To: References: <303a8c7b-1c1b-12c0-bb0c-4cd70be8611a@kynesim.co.uk> <6ff313c1-d38b-5c5b-99cc-743413416f2f@kynesim.co.uk> Message-ID: Thanks to all giving me helpful comments. Multiple hexdump files are found. Maybe that"s the reason. I guess i can figure out On Thu, Dec 19, 2019 at 5:29 PM Dennis Lee Bieber wrote: > On Thu, 19 Dec 2019 12:43:11 +0000, tommy yama > declaimed the following: > > > > >user at USERnoMacBook-Air LibraBrowser % python3 hexdump.py > > > > >/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS/Python: > >can't open file 'hexdump.py': [Errno 2] No such file or directory > > > >user at USERnoMacBook-Air LibraBrowser % > > > > Well... what does > > find / -iname hexdump.py > > return? If found, is it in the Python search path? > > {Debian system here, but... (after doing apt install python3-pip)} > > wulfraed at debian:~$ pip3 install hexdump > Collecting hexdump > Downloading > > https://files.pythonhosted.org/packages/55/b3/279b1d57fa3681725d0db8820405cdcb4e62a9239c205e4ceac4391c78e4/hexdump-3.3.zip > Building wheels for collected packages: hexdump > Running setup.py bdist_wheel for hexdump ... done > Stored in directory: > > /home/wulfraed/.cache/pip/wheels/d5/d1/f2/c8183b5863b3df595c2eeafd8e015a43dae13d403a959467c6 > Successfully built hexdump > Installing collected packages: hexdump > Successfully installed hexdump-3.3 > wulfraed at debian:~$ sudo find / -iname hexdump.py > /home/wulfraed/.local/lib/python3.7/site-packages/hexdump.py > find: ?/run/user/1000/gvfs?: Permission denied > /usr/lib/python3/dist-packages/pygments/lexers/hexdump.py > /usr/lib/python2.7/dist-packages/pygments/lexers/hexdump.py > wulfraed at debian:~$ python3 > Python 3.7.3 (default, Apr 3 2019, 05:39:12) > [GCC 8.3.0] on linux > Type "help", "copyright", "credits" or "license" for more information. > >>> import hexdump > >>> dir(hexdump) > ['PY3K', '__author__', '__builtins__', '__cached__', '__doc__', '__file__', > '__history__', '__license__', '__loader__', '__name__', '__package__', > '__spec__', '__version__', 'binascii', 'chunkread', 'chunks', 'dehex', > 'dump', 'dumpgen', 'genchunks', 'hexdump', 'main', 'normalize_py', > 'restore', 'runtest', 'sys'] > >>> hexdump.__file__ > '/home/wulfraed/.local/lib/python3.7/site-packages/hexdump.py' > >>> > > Note that there are "hexdump.py" files involved with the pygments > package, which likely are not what you are looking for (quick glance -- the > pygments hexdump READS dump files). > > > -- > Wulfraed Dennis Lee Bieber AF6VN > wlfraed at ix.netcom.com > http://wlfraed.microdiversity.freeddns.org/ > > -- > https://mail.python.org/mailman/listinfo/python-list > From barry at barrys-emacs.org Sat Dec 21 04:58:42 2019 From: barry at barrys-emacs.org (Barry) Date: Sat, 21 Dec 2019 09:58:42 +0000 Subject: Problems with "Tarfile.close()" In-Reply-To: <24060.48206.298944.886236@woitok.gmail.com> References: <24060.48206.298944.886236@woitok.gmail.com> Message-ID: <673433FC-434D-4AE2-83CE-B292F0584C38@barrys-emacs.org> > On 20 Dec 2019, at 15:31, Dr Rainer Woitok wrote: > > ?Greetings, > > One of my Python scripts basically does the following: > > source = tarfile.open(name=tar_archive , mode='r|*') > dest = tarfile.open(fileobj=sys.stdout, mode='w|', format=fmt) > > . > . > . > > source.close() > dest.close() > > In an attempt to move my Python scripts from Python 2.7 to Python 3.6 I > ran into the problem that under Python 3.6 the call to "dest.close()" > fails: > > Traceback (most recent call last): > File ".../tar_archive.copy", line 137, in > dest.close() > File "/usr/lib64/python3.6/tarfile.py", line 1742, in close > self.fileobj.close() > File "/usr/lib64/python3.6/tarfile.py", line 467, in close > self.fileobj.write(self.buf) > TypeError: write() argument must be str, not bytes > > What am I doing wrong? By the way: since on some hosts this script is > running on the transition from Python 2.7 to Python 3.x will not happen > immediately, I need a solution which works with both versions. Stdout on python 3 is a text stream. Tarfiles need a binary stream. I searched for ?python 3 binary stdout? and found this stack over flow question that has suggested fixes. https://stackoverflow.com/questions/908331/how-to-write-binary-data-to-stdout-in-python-3#908440 Barry > > Sincerely, > Rainer > -- > https://mail.python.org/mailman/listinfo/python-list > From barry at barrys-emacs.org Sat Dec 21 05:05:22 2019 From: barry at barrys-emacs.org (Barry) Date: Sat, 21 Dec 2019 10:05:22 +0000 Subject: Understanding of GIL In-Reply-To: References: Message-ID: <919498FD-709C-4C8A-B87E-AA2B8C2789DF@barrys-emacs.org> > On 19 Dec 2019, at 14:19, onlinejudge95 wrote: > > ?Hi Devs, > > I am currently writing some custom Django commands for data updation, my > workflow is like > > Fetch data from *PostgreSQL*. > Call *Elasticsearch* for searching based on the data fetched from > *PostgreSQL*. > Query *PostgreSQL* and do an upsert behavior. > I am using pandas data frame to hold my data during processing. > > The host we are using to run this jobs has a *CPython* interpreter as given > by > > `platform.python_implementation()` > > I want to confirm whether *multithreading* would be a better choice here, > given the fact that GIL is the biggest blocker(I agree it has to be there) > for the same in CPython interpreters. You mean multi process vs. Threaded? The GIL is only a problem if you are executing pure python compute bound code on multiple threads. When your code does lots of I/O and uses extensions it?s not the limiting factor. I guess you app will spent if time waiting of PostgreSQL etc. Barry > > In case further information is required do let me know. > > Thanks > onlinejudge95 > -- > https://mail.python.org/mailman/listinfo/python-list > From bluebox03 at gmail.com Sat Dec 21 08:55:05 2019 From: bluebox03 at gmail.com (tommy yama) Date: Sat, 21 Dec 2019 13:55:05 +0000 Subject: hexdump module installation error In-Reply-To: References: <303a8c7b-1c1b-12c0-bb0c-4cd70be8611a@kynesim.co.uk> <6ff313c1-d38b-5c5b-99cc-743413416f2f@kynesim.co.uk> Message-ID: fixed now. I guess pip3 is linked to pip2 and installed python3 again. On Sat, Dec 21, 2019 at 8:44 AM tommy yama wrote: > Thanks to all giving me helpful comments. > Multiple hexdump files are found. Maybe that"s the reason. > > I guess i can figure out > > > > > On Thu, Dec 19, 2019 at 5:29 PM Dennis Lee Bieber > wrote: > >> On Thu, 19 Dec 2019 12:43:11 +0000, tommy yama >> declaimed the following: >> >> > >> >user at USERnoMacBook-Air LibraBrowser % python3 hexdump.py >> > >> >> >/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS/Python: >> >can't open file 'hexdump.py': [Errno 2] No such file or directory >> > >> >user at USERnoMacBook-Air LibraBrowser % >> > >> >> Well... what does >> >> find / -iname hexdump.py >> >> return? If found, is it in the Python search path? >> >> {Debian system here, but... (after doing apt install python3-pip)} >> >> wulfraed at debian:~$ pip3 install hexdump >> Collecting hexdump >> Downloading >> >> https://files.pythonhosted.org/packages/55/b3/279b1d57fa3681725d0db8820405cdcb4e62a9239c205e4ceac4391c78e4/hexdump-3.3.zip >> Building wheels for collected packages: hexdump >> Running setup.py bdist_wheel for hexdump ... done >> Stored in directory: >> >> /home/wulfraed/.cache/pip/wheels/d5/d1/f2/c8183b5863b3df595c2eeafd8e015a43dae13d403a959467c6 >> Successfully built hexdump >> Installing collected packages: hexdump >> Successfully installed hexdump-3.3 >> wulfraed at debian:~$ sudo find / -iname hexdump.py >> /home/wulfraed/.local/lib/python3.7/site-packages/hexdump.py >> find: ?/run/user/1000/gvfs?: Permission denied >> /usr/lib/python3/dist-packages/pygments/lexers/hexdump.py >> /usr/lib/python2.7/dist-packages/pygments/lexers/hexdump.py >> wulfraed at debian:~$ python3 >> Python 3.7.3 (default, Apr 3 2019, 05:39:12) >> [GCC 8.3.0] on linux >> Type "help", "copyright", "credits" or "license" for more information. >> >>> import hexdump >> >>> dir(hexdump) >> ['PY3K', '__author__', '__builtins__', '__cached__', '__doc__', >> '__file__', >> '__history__', '__license__', '__loader__', '__name__', '__package__', >> '__spec__', '__version__', 'binascii', 'chunkread', 'chunks', 'dehex', >> 'dump', 'dumpgen', 'genchunks', 'hexdump', 'main', 'normalize_py', >> 'restore', 'runtest', 'sys'] >> >>> hexdump.__file__ >> '/home/wulfraed/.local/lib/python3.7/site-packages/hexdump.py' >> >>> >> >> Note that there are "hexdump.py" files involved with the pygments >> package, which likely are not what you are looking for (quick glance -- >> the >> pygments hexdump READS dump files). >> >> >> -- >> Wulfraed Dennis Lee Bieber AF6VN >> wlfraed at ix.netcom.com >> http://wlfraed.microdiversity.freeddns.org/ >> >> -- >> https://mail.python.org/mailman/listinfo/python-list >> > From john_ladasky at sbcglobal.net Sat Dec 21 14:16:59 2019 From: john_ladasky at sbcglobal.net (John Ladasky) Date: Sat, 21 Dec 2019 11:16:59 -0800 (PST) Subject: How to specific multiple dtypes in numpy.ndarray? In-Reply-To: References: Message-ID: <38991919-f1c9-42c4-9b49-55b8e002572a@googlegroups.com> On Thursday, December 19, 2019 at 2:53:18 AM UTC-8, lampahome wrote: > I meet performance is low when I use struct.unpack to unpack binary data. > > So I tried to use numpy.ndarray > But meet error when I want to unpack multiple dtypes > > Can anyone teach me~ > > Code like below: > # python3 > import struct > import numpy as np > s1 = struct.Struct("@QIQ") > ss1 = s1.pack(1,11,111) > np.ndarray((3,), [('Q','I','Q')], ss1) > # ValueError: mismatch in size of old and new data-descriptor. Is there some reason that you're calling the np.ndarray constructor directly instead of the np.array convenience function? The arguments that you pass to each are slightly different. Generally, you want to use np.array. The first argument to np.ndarray is the shape. You appear to want three elements. The second argument to np.ndarray is the data type (dtype). It needs to be a SINGLE data type, like 'float' or 'int' or 'object'. All elements in in array are of the same dtype. If you want different behavior, I think that you want lists, not arrays. You can set the dtype to 'object' and you should be able to store any data type as an array element. But I don't think that offers you any advantages over using ordinary Python lists. In any case, it appears that you aren't passing a dtype as your second argument. You're trying to pass the data itself, as a tuple wrapped in a single list. This seems to be certain to raise an error. The third argument to np.ndarray is supposed to be the buffer which contains your data. https://docs.scipy.org/doc/numpy/reference/generated/numpy.array.html#numpy.array ile:///usr/share/doc/python-numpy-doc/html/reference/generated/numpy.array.html#numpy.array From bill at baddogconsulting.com Sat Dec 21 15:57:24 2019 From: bill at baddogconsulting.com (Bill Deegan) Date: Sat, 21 Dec 2019 12:57:24 -0800 Subject: [Scons-dev] SCons 3.1.2 Released In-Reply-To: References: Message-ID: Glad that helped! Big thanks to Mats Wichmann for that one! On Sat, Dec 21, 2019 at 10:41 AM Eric Fahlgren wrote: > On Mon, Dec 16, 2019 at 7:02 PM Bill Deegan > wrote: > >> - EXPERIMENTAL NEW FEATURE: Enable caching MSVC configuration >> If SCONS_CACHE_MSVC_CONFIG shell environment variable is set, >> SCons will cache the results of past calls to vcvarsall.bat to >> a file; integrates with existing memoizing of such vars. >> On vs2019 saves 5+ seconds per SCons invocation, which really >> helps test suite runs. >> > > Thanks a bunch for this one! Our bare, no-target 'scons' invocation is > down to about 0.8 seconds now, and we can do a single DLL build in ~2.5 s, > which makes things very snappy indeed. > _______________________________________________ > Scons-dev mailing list > Scons-dev at scons.org > https://pairlist2.pair.net/mailman/listinfo/scons-dev > From benandrewhearn at gmail.com Sat Dec 21 16:46:31 2019 From: benandrewhearn at gmail.com (Ben Hearn) Date: Sat, 21 Dec 2019 13:46:31 -0800 (PST) Subject: urllib unqoute providing string mismatch between string found using os.walk (Python3) Message-ID: <2c2f5420-00ec-483e-95bb-0b5a1bcde363@googlegroups.com> Hello all, I am having a bit of trouble with a string mismatch operation in my tool I am writing. I am comparing a database collection or url quoted paths to the paths on the users drive. These 2 paths look identical, one from the drive & the other from an xml url: a = '/Users/macbookpro/Music/tracks_new/_NS_2018/J.Staaf - ?Mo?chate! _PromoMix_.wav' b = '/Users/macbookpro/Music/tracks_new/_NS_2018/J.Staaf - ?M?chate! _PromoMix_.wav' But after realising it was failing on them I ran a difflib and these differences popped up. import difflib print('\n'.join(difflib.ndiff([a], [b]))) - /Users/macbookpro/Music/tracks_new/_NS_2018/J.Staaf - ?Mo?chate! _PromoMix_.wav ? ^^ + /Users/macbookpro/Music/tracks_new/_NS_2018/J.Staaf - ?M?chate! _PromoMix_.wav ? ^ What am I missing when it comes to unquoting the string, or should I do some other fancy operation on the drive string? Cheers, Ben From pieter-l at vanoostrum.org Sat Dec 21 17:23:17 2019 From: pieter-l at vanoostrum.org (Pieter van Oostrum) Date: Sat, 21 Dec 2019 23:23:17 +0100 Subject: urllib unqoute providing string mismatch between string found using os.walk (Python3) References: <2c2f5420-00ec-483e-95bb-0b5a1bcde363@googlegroups.com> Message-ID: Ben Hearn writes: > Hello all, > > I am having a bit of trouble with a string mismatch operation in my tool I am writing. > > I am comparing a database collection or url quoted paths to the paths on the users drive. > > These 2 paths look identical, one from the drive & the other from an xml url: > a = '/Users/macbookpro/Music/tracks_new/_NS_2018/J.Staaf - ?Mo?chate! _PromoMix_.wav' > b = '/Users/macbookpro/Music/tracks_new/_NS_2018/J.Staaf - ?M?chate! _PromoMix_.wav' > > But after realising it was failing on them I ran a difflib and these differences popped up. > > import difflib > print('\n'.join(difflib.ndiff([a], [b]))) > - /Users/macbookpro/Music/tracks_new/_NS_2018/J.Staaf - ?Mo?chate! _PromoMix_.wav > ? ^^ > + /Users/macbookpro/Music/tracks_new/_NS_2018/J.Staaf - ?M?chate! _PromoMix_.wav > ? ^ > > > What am I missing when it comes to unquoting the string, or should I do > some other fancy operation on the drive string? > In [8]: len(a) Out[8]: 79 In [9]: len(b) Out[9]: 78 The difference is in the ?. In (b) it is a single character, Unicode 0xF3, LATIN SMALL LETTER O WITH ACUTE. In (a) it is composed of the letter o and the accent "?" (Unicode 0x301). So you would have to do Unicode normalisation before comparing. For example: In [16]: from unicodedata import normalize In [17]: a == b Out[17]: False In [18]: normalize('NFC', a) == normalize('NFC', b) Out[18]: True -- Pieter van Oostrum www: http://pieter.vanoostrum.org/ PGP key: [8DAE142BE17999C4] From 2QdxY4RzWzUUiLuE at potatochowder.com Sat Dec 21 17:15:47 2019 From: 2QdxY4RzWzUUiLuE at potatochowder.com (Dan Sommers) Date: Sat, 21 Dec 2019 17:15:47 -0500 Subject: urllib unqoute providing string mismatch between string found using os.walk (Python3) In-Reply-To: <2c2f5420-00ec-483e-95bb-0b5a1bcde363@googlegroups.com> References: <2c2f5420-00ec-483e-95bb-0b5a1bcde363@googlegroups.com> Message-ID: On 12/21/19 4:46 PM, Ben Hearn wrote: > import difflib > print('\n'.join(difflib.ndiff([a], [b]))) > - /Users/macbookpro/Music/tracks_new/_NS_2018/J.Staaf - ?Mo?chate! _PromoMix_.wav > ? ^^ > + /Users/macbookpro/Music/tracks_new/_NS_2018/J.Staaf - ?M?chate! _PromoMix_.wav > ? ^ > > > What am I missing when it comes to unquoting the string, or should I do some other fancy operation on the drive string? I'm going to guess that the trailing characters are newline (U+0010) and/or carriage return (U+001D) characters. Use str.strip() to remove them before comparing: a = a.strip() Dan From ericfahlgren at gmail.com Sat Dec 21 13:41:34 2019 From: ericfahlgren at gmail.com (Eric Fahlgren) Date: Sat, 21 Dec 2019 10:41:34 -0800 Subject: [Scons-dev] SCons 3.1.2 Released In-Reply-To: References: Message-ID: On Mon, Dec 16, 2019 at 7:02 PM Bill Deegan wrote: > - EXPERIMENTAL NEW FEATURE: Enable caching MSVC configuration > If SCONS_CACHE_MSVC_CONFIG shell environment variable is set, > SCons will cache the results of past calls to vcvarsall.bat to > a file; integrates with existing memoizing of such vars. > On vs2019 saves 5+ seconds per SCons invocation, which really > helps test suite runs. > Thanks a bunch for this one! Our bare, no-target 'scons' invocation is down to about 0.8 seconds now, and we can do a single DLL build in ~2.5 s, which makes things very snappy indeed. From torriem at gmail.com Sat Dec 21 19:22:55 2019 From: torriem at gmail.com (Michael Torrie) Date: Sat, 21 Dec 2019 17:22:55 -0700 Subject: urllib unqoute providing string mismatch between string found using os.walk (Python3) In-Reply-To: <2c2f5420-00ec-483e-95bb-0b5a1bcde363@googlegroups.com> References: <2c2f5420-00ec-483e-95bb-0b5a1bcde363@googlegroups.com> Message-ID: <451e70f7-187a-684b-789d-dd35060d69bb@gmail.com> On 12/21/19 2:46 PM, Ben Hearn wrote: > These 2 paths look identical, one from the drive & the other from an xml url: > a = '/Users/macbookpro/Music/tracks_new/_NS_2018/J.Staaf - ?Mo?chate! _PromoMix_.wav' ^^ > b = '/Users/macbookpro/Music/tracks_new/_NS_2018/J.Staaf - ?M?chate! _PromoMix_.wav' ^^ They are actually are different strings. The name is spelled differently between the two. Mo?chate vs M?chate (the former seems to be the correct spelling according to my inline spell checker). Is this from your own program? I wonder how it got switched? From rosuav at gmail.com Sat Dec 21 20:14:01 2019 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 22 Dec 2019 12:14:01 +1100 Subject: urllib unqoute providing string mismatch between string found using os.walk (Python3) In-Reply-To: <451e70f7-187a-684b-789d-dd35060d69bb@gmail.com> References: <2c2f5420-00ec-483e-95bb-0b5a1bcde363@googlegroups.com> <451e70f7-187a-684b-789d-dd35060d69bb@gmail.com> Message-ID: On Sun, Dec 22, 2019 at 11:33 AM Michael Torrie wrote: > > On 12/21/19 2:46 PM, Ben Hearn wrote: > > These 2 paths look identical, one from the drive & the other from an xml url: > > a = '/Users/macbookpro/Music/tracks_new/_NS_2018/J.Staaf - ?Mo?chate! _PromoMix_.wav' > ^^ > > b = '/Users/macbookpro/Music/tracks_new/_NS_2018/J.Staaf - ?M?chate! _PromoMix_.wav' > ^^ > They are actually are different strings. The name is spelled > differently between the two. Mo?chate vs M?chate (the former seems to > be the correct spelling according to my inline spell checker). Is this > from your own program? I wonder how it got switched? Then your inline spell checker is flawed, because the two strings differ only in whether a single "o with acute" or a pair "o" and "combining acute" is used to represent the accented letter. The word is the same either way. ChrisA From python at mrabarnett.plus.com Sat Dec 21 20:25:11 2019 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 22 Dec 2019 01:25:11 +0000 Subject: urllib unqoute providing string mismatch between string found using os.walk (Python3) In-Reply-To: <451e70f7-187a-684b-789d-dd35060d69bb@gmail.com> References: <2c2f5420-00ec-483e-95bb-0b5a1bcde363@googlegroups.com> <451e70f7-187a-684b-789d-dd35060d69bb@gmail.com> Message-ID: On 2019-12-22 00:22, Michael Torrie wrote: > On 12/21/19 2:46 PM, Ben Hearn wrote: >> These 2 paths look identical, one from the drive & the other from an xml url: >> a = '/Users/macbookpro/Music/tracks_new/_NS_2018/J.Staaf - ?Mo?chate! _PromoMix_.wav' > ^^ >> b = '/Users/macbookpro/Music/tracks_new/_NS_2018/J.Staaf - ?M?chate! _PromoMix_.wav' > ^^ > They are actually are different strings. The name is spelled > differently between the two. Mo?chate vs M?chate (the former seems to > be the correct spelling according to my inline spell checker). Is this > from your own program? I wonder how it got switched? > Use the 'ascii' function to see what's different: >>> ascii(a) "'/Users/macbookpro/Music/tracks_new/_NS_2018/J.Staaf - \\xa1Mo\\u0301chate! _PromoMix_.wav'" >>> ascii(b) "'/Users/macbookpro/Music/tracks_new/_NS_2018/J.Staaf - \\xa1M\\xf3chate! _PromoMix_.wav'" >>> From Richard at Damon-Family.org Sat Dec 21 21:36:16 2019 From: Richard at Damon-Family.org (Richard Damon) Date: Sat, 21 Dec 2019 21:36:16 -0500 Subject: urllib unqoute providing string mismatch between string found using os.walk (Python3) In-Reply-To: References: <2c2f5420-00ec-483e-95bb-0b5a1bcde363@googlegroups.com> <451e70f7-187a-684b-789d-dd35060d69bb@gmail.com> Message-ID: <3c2b29f3-9654-a5a1-b328-e8e79812a62e@Damon-Family.org> On 12/21/19 8:25 PM, MRAB wrote: > On 2019-12-22 00:22, Michael Torrie wrote: >> On 12/21/19 2:46 PM, Ben Hearn wrote: >>> These 2 paths look identical, one from the drive & the other from an >>> xml url: >>> a = '/Users/macbookpro/Music/tracks_new/_NS_2018/J.Staaf - >>> ?Mo?chate! _PromoMix_.wav' >> ??????????????????????????????????????????????????????????????? ^^ >>> b = '/Users/macbookpro/Music/tracks_new/_NS_2018/J.Staaf - ?M?chate! >>> _PromoMix_.wav' >> ??????????????????????????????????????????????????????????????? ^^ >> They are actually are different strings.? The name is spelled >> differently between the two.? Mo?chate vs M?chate (the former seems to >> be the correct spelling according to my inline spell checker).? Is this >> from your own program? I wonder how it got switched? >> > Use the 'ascii' function to see what's different: > > >>> ascii(a) > "'/Users/macbookpro/Music/tracks_new/_NS_2018/J.Staaf - > \\xa1Mo\\u0301chate! _PromoMix_.wav'" > >>> ascii(b) > "'/Users/macbookpro/Music/tracks_new/_NS_2018/J.Staaf - > \\xa1M\\xf3chate! _PromoMix_.wav'" > >>> It is a Unicode Normalization issue. A number of characters can be 'spelled' different ways. ? can be either a single codepoint U+00F3, or it can be the pair of codepoints, the o and U+0301 (the accent). If you want to make the strings compare equal then you need to make sure that you have normalized both strings the same way. I beleive that the Mac OS always converts file names into the NFD format when it uses them (that is what the first (a) string is in) -- Richard Damon From nt_mahmood at yahoo.com Sun Dec 22 09:07:36 2019 From: nt_mahmood at yahoo.com (Mahmood Naderan) Date: Sun, 22 Dec 2019 14:07:36 +0000 (UTC) Subject: Unable to install "collect" via pip3 In-Reply-To: References: <414413727.1095151.1576826520979@mail.yahoo.com> Message-ID: <31468466.1581608.1577023656377@mail.yahoo.com> Yes thank you. The package is not compatible with 3.x. Regards, Mahmood On Saturday, December 21, 2019, 1:40:29 AM GMT+3:30, Barry wrote: > On 20 Dec 2019, at 15:27, Mahmood Naderan via Python-list wrote: > > ?Hi > > I can install collect with pip for python2.7 > $ pip install --user collect? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? > Collecting collect >? Using cached https://files.pythonhosted.org/packages/cf/5e/c0f0f51d081665374a2c219ea4ba23fb1e179b70dded96dc16606786d828/collect-0.1.1.tar.gz > Collecting couchdbkit>=0.5.7 (from collect) >? Using cached https://files.pythonhosted.org/packages/a1/13/9e9ff695a385c44f62b4766341b97f2bd8b596962df2a0beabf358468b70/couchdbkit-0.6.5.tar.gz > Collecting restkit>=4.2.2 (from couchdbkit>=0.5.7->collect) >? Downloading https://files.pythonhosted.org/packages/76/b9/d90120add1be718f853c53008cf5b62d74abad1d32bd1e7097dd913ae053/restkit-4.2.2.tar.gz (1.3MB) >? ? 100% |????????????????????????????????| 1.3MB 633kB/s > Collecting http-parser>=0.8.3 (from restkit>=4.2.2->couchdbkit>=0.5.7->collect) >? Downloading https://files.pythonhosted.org/packages/07/c4/22e3c76c2313c26dd5f84f1205b916ff38ea951aab0c4544b6e2f5920d64/http-parser-0.8.3.tar.gz (83kB) >? ? 100% |????????????????????????????????| 92kB 2.4MB/s > Collecting socketpool>=0.5.3 (from restkit>=4.2.2->couchdbkit>=0.5.7->collect) >? Downloading https://files.pythonhosted.org/packages/d1/39/fae99a735227234ffec389b252c6de2bc7816bf627f56b4c558dc46c85aa/socketpool-0.5.3.tar.gz > Building wheels for collected packages: collect, couchdbkit, restkit, http-parser, socketpool >? Running setup.py bdist_wheel for collect ... done >? Stored in directory: /home/mnaderan/.cache/pip/wheels/b9/7c/7c/b09b334cc0e27b4f63ee9f6f19ca1f3db8672666a7e0f3d9cd >? Running setup.py bdist_wheel for couchdbkit ... done >? Stored in directory: /home/mnaderan/.cache/pip/wheels/f6/05/1b/f8f576ef18564bc68ab6e64f405e1263448036208cafb221e0 >? Running setup.py bdist_wheel for restkit ... done >? Stored in directory: /home/mnaderan/.cache/pip/wheels/48/c5/32/d0d25fb272791a68c49c26150f332d9b9492d0bc9ea0cdd2c7 >? Running setup.py bdist_wheel for http-parser ... done >? Stored in directory: /home/mnaderan/.cache/pip/wheels/22/db/06/cb609a3345e7aa87206de160f00cc6af364650d1139d904a25 >? Running setup.py bdist_wheel for socketpool ... done >? Stored in directory: /home/mnaderan/.cache/pip/wheels/93/f6/8c/65924848766618647078cb66b1d964e8b80876536e84517469 > Successfully built collect couchdbkit restkit http-parser socketpool > Installing collected packages: http-parser, socketpool, restkit, couchdbkit, collect > Successfully installed collect-0.1.1 couchdbkit-0.6.5 http-parser-0.8.3 restkit-4.2.2 socketpool-0.5.3 > However, pip3 fails with this error > $ pip3 install --user collect > Collecting collect >? Using cached https://files.pythonhosted.org/packages/cf/5e/c0f0f51d081665374a2c219ea4ba23fb1e179b70dded96dc16606786d828/collect-0.1.1.tar.gz > Collecting couchdbkit>=0.5.7 (from collect) >? Using cached https://files.pythonhosted.org/packages/a1/13/9e9ff695a385c44f62b4766341b97f2bd8b596962df2a0beabf358468b70/couchdbkit-0.6.5.tar.gz >? ? Complete output from command python setup.py egg_info: >? ? Traceback (most recent call last): >? ? ? File "", line 1, in >? ? ? File "/tmp/pip-build-qf95n0tt/couchdbkit/setup.py", line 25, in >? ? ? ? long_description = file( >? ? NameError: name 'file' is not defined My guess is that file is python 2 only. Couchdbkit needs porting to python 3. Barry > >? ? ---------------------------------------- > Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-qf95n0tt/couchdbkit/ > I can not figure out what is the problem. Any way to fix that? > > More info: > $ which python > /usr/bin/python > $ ls -l /usr/bin/python > lrwxrwxrwx 1 root root 9 Apr 16? 2018 /usr/bin/python -> python2.7 > $ which python3 > /usr/bin/python3 > $ ls -l /usr/bin/python3 > lrwxrwxrwx 1 root root 9 Jun 21? 2018 /usr/bin/python3 -> python3.6 > > > > Regards, > Mahmood > -- > https://mail.python.org/mailman/listinfo/python-list From benandrewhearn at gmail.com Sun Dec 22 10:38:08 2019 From: benandrewhearn at gmail.com (Ben Hearn) Date: Sun, 22 Dec 2019 07:38:08 -0800 (PST) Subject: urllib unqoute providing string mismatch between string found using os.walk (Python3) In-Reply-To: <2c2f5420-00ec-483e-95bb-0b5a1bcde363@googlegroups.com> References: <2c2f5420-00ec-483e-95bb-0b5a1bcde363@googlegroups.com> Message-ID: <87c26489-3185-446c-8dc1-14f78c83e6a2@googlegroups.com> On Saturday, 21 December 2019 21:46:43 UTC, Ben Hearn wrote: > Hello all, > > I am having a bit of trouble with a string mismatch operation in my tool I am writing. > > I am comparing a database collection or url quoted paths to the paths on the users drive. > > These 2 paths look identical, one from the drive & the other from an xml url: > a = '/Users/macbookpro/Music/tracks_new/_NS_2018/J.Staaf - ?Mo?chate! _PromoMix_.wav' > b = '/Users/macbookpro/Music/tracks_new/_NS_2018/J.Staaf - ?M?chate! _PromoMix_.wav' > > But after realising it was failing on them I ran a difflib and these differences popped up. > > import difflib > print('\n'.join(difflib.ndiff([a], [b]))) > - /Users/macbookpro/Music/tracks_new/_NS_2018/J.Staaf - ?Mo?chate! _PromoMix_.wav > ? ^^ > + /Users/macbookpro/Music/tracks_new/_NS_2018/J.Staaf - ?M?chate! _PromoMix_.wav > ? ^ > > > What am I missing when it comes to unquoting the string, or should I do some other fancy operation on the drive string? > > Cheers, > > Ben Thanks for the help on this, normalizing my string fixed up the problem cheers! From wojtek.dominicana at gmail.com Sun Dec 22 10:40:09 2019 From: wojtek.dominicana at gmail.com (wojtek.dominicana at gmail.com) Date: Sun, 22 Dec 2019 07:40:09 -0800 (PST) Subject: Execute python script from VB6 to display Outlook Window Message-ID: I have to prepare e-mail in python (e-mail address, body html, attachments etc.) and show Outlook window for user who will check everything and correct if necessary and will be able to send it. My script work well : o = win32com.client.Dispatch("Outlook.Application") Msg = o.CreateItem(0) Msg.To = "aa at gmail.com" Msg.CC = "" Msg.Subject = "The random subject" Msg.HtmlBody = "

content of e-mail

" Msg.Attachments.Add(attachment1) Msg.Display() This script work well but I have some problem with my Outlook 2019 or Command window (Win 10, 64). When Outlook is closed i can call my script using Shell() or ShellExecute() and everything is ok but when MS Outlook is open my script doesnt work. I use the same Shell() or ShellExecute() function and still does not work although windows executing this script. It stops in the loine "win32com.client.Dispatch("Outlook.Application")". I tried many tricks and I gave up. The most important is that when I use windows command line and run my script manyally (python sendoutlookmail.py) everything works well regardless of state of Outlook (open or closed). I think there is a different between calling script via Shell() function and run it directly from command line. I tried of course solution as follows: Set obj = CreateObject("WScript.Shell") obj.Run sPythonFile but its the same situation. I createt bat file and I execute this bat file which has inside commant to execute my python script. I have no idea how to run python script on my system that works the same as I call it directly from windows command line. regards Wojtek From musbur at posteo.org Sun Dec 22 12:18:38 2019 From: musbur at posteo.org (musbur at posteo.org) Date: Sun, 22 Dec 2019 18:18:38 +0100 Subject: Transfer a file to httpserver via POST command from curl In-Reply-To: References: <20191217184404.298104a4@nxp10225> Message-ID: <20191222181838.3918384c@nxp10225> On Wed, 18 Dec 2019 04:52:33 +1100 Chris Angelico wrote: > On Wed, Dec 18, 2019 at 4:45 AM wrote: > > BTW, the canonical way to upload files via http is PUT, not POST. > > You might want to look into that, but here it is off-topic. > > Citation needed. https://tools.ietf.org/html/rfc2616#page-55 > Plenty of file uploads are done through POST > requests. Of course. Both work. It's just that the OP wanted to "upload a large binary file" using curl and in such cases I find that PUT can make for a cleaner, simpler interface. > Are you talking specifically about a RESTful API? Because > that's only one of many patterns you can follow. Sure. It's just that people sometimes aren't even aware of http methods besides GET and POST, and there's a chance for the OP to investigate this and maybe to find that PUT fits his needs better than POST in this case. From rosuav at gmail.com Sun Dec 22 12:24:22 2019 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 23 Dec 2019 04:24:22 +1100 Subject: Transfer a file to httpserver via POST command from curl In-Reply-To: <20191222181838.3918384c@nxp10225> References: <20191217184404.298104a4@nxp10225> <20191222181838.3918384c@nxp10225> Message-ID: On Mon, Dec 23, 2019 at 4:20 AM wrote: > > On Wed, 18 Dec 2019 04:52:33 +1100 > Chris Angelico wrote: > > > On Wed, Dec 18, 2019 at 4:45 AM wrote: > > > BTW, the canonical way to upload files via http is PUT, not POST. > > > You might want to look into that, but here it is off-topic. > > > > Citation needed. > > https://tools.ietf.org/html/rfc2616#page-55 Yes, that's the definition of PUT - but what does that mean in terms of file uploads? POST is also valid. Look at its example usages. > > Plenty of file uploads are done through POST > > requests. > > Of course. Both work. It's just that the OP wanted to "upload a large > binary file" using curl and in such cases I find that PUT can make for > a cleaner, simpler interface. Given that curl lets you specify any method for a request, I doubt that PUT is going to be any cleaner or simpler. > > Are you talking specifically about a RESTful API? Because > > that's only one of many patterns you can follow. > > Sure. It's just that people sometimes aren't even aware of http > methods besides GET and POST, and there's a chance for the OP > to investigate this and maybe to find that PUT fits his needs better > than POST in this case. True, it's worth knowing about other HTTP methods. But stating that PUT is "the canonical way to upload files" implies that what the OP is doing is somehow violating the spec, which simply isn't true. ChrisA From PythonList at DancesWithMice.info Sun Dec 22 14:35:50 2019 From: PythonList at DancesWithMice.info (DL Neil) Date: Mon, 23 Dec 2019 08:35:50 +1300 Subject: More efficient/elegant branching In-Reply-To: References: <4c0405e8fe1ca0af482048c60e9ee2f4@posteo.de> Message-ID: <9823ad20-6c92-6606-6268-845ec144c5cf@DancesWithMice.info> On 11/12/19 1:07 AM, Daniel Haude wrote: > Hello Neil, > thanks for the detailed answer. > >> Question: are there other people/factors who/which should be regarded >> as more important than the linter's opinion? > Yes. Mine. Um, see below... (unless humor) > I was just puzzled at the linter's output (took me a while to figure out > what it actually meant), and that got me started on the track if there > was maybe a more Pythonic way of dealing with that decision chain. ... >> One reason why this code looks a bit strange (and possibly why PyLint >> reacts?) is because it is trivial. When we look at the overall >> picture, the question becomes: what will the 'mainline' do with >> "result" (the returned value)? Immediately one suspects it will be fed >> into another decision, eg: > > No, the "result" is a text message that actually means something and is > eventually output for human consumption. > > The whole thing is rather academic. Also my efficiency argument doesn't > hold water because this routine is executed just a few times per hour. I > like the "condition table" approach for its lower line count but I'll > stick with if/elif because it's more conventional and therefore easier > to understand for the casual reader. +1 (except that it's your future-self or another maintainer who is important rather than anyone 'casual') A reason we are having difficulty making suggestions is because the example has been abstracted. Thus we had no idea that the objective was to (possibly) output to SMS, nor more importantly, the meanings of a, b, and z. (it's all algebra to me!) In some/another application you may like to consider constructing the source-data for this decision within a class. Then the conditions can become methods or even properties of the class - remembering that a property could be some combination of attributes, eg a > 4 and b == 0. Alternatively, the class could maintain some idea of 'status', where "first" corresponds to the preceding 'state', "second" when len(z) falls below 2, etc. The class's property/ies thus simplify the SMS-transmission process/class... Appropriate name-choices will make the code self-documenting - per the 'reading' objective: if employee.worked_overtime: sms.send( "We're in the money...!" ) -- Regards =dn From isidentical at gmail.com Sun Dec 22 15:34:46 2019 From: isidentical at gmail.com (Batuhan Taskaya) Date: Sun, 22 Dec 2019 23:34:46 +0300 Subject: Most elegant way to do something N times Message-ID: I encounter with cases like doing a function 6 time with no argument, or same arguments over and over or doing some structral thing N times and I dont know how elegant I can express that to the code. I dont know why but I dont like this for _ in range(n): do() thing. Any suggestions? From Richard at Damon-Family.org Sun Dec 22 16:23:31 2019 From: Richard at Damon-Family.org (Richard Damon) Date: Sun, 22 Dec 2019 16:23:31 -0500 Subject: Most elegant way to do something N times In-Reply-To: References: Message-ID: <97245842-171f-4e20-20b4-8dd31cd62ca1@Damon-Family.org> On 12/22/19 3:34 PM, Batuhan Taskaya wrote: > I encounter with cases like doing a function 6 time with no argument, or > same arguments over and over or doing some structral thing N times and I > dont know how elegant I can express that to the code. I dont know why but I > dont like this for _ in range(n): do() thing. Any suggestions? For me, the statement for _ in range(n): do() is the most straight forward and clearest expression of that idea. If you don't like the _, you can always use some 'worthless' variable to get rid of the special _ syntax. You could of course use something like a while loop to build this, but in my mind that is just making things less clear. -- Richard Damon From mirkok.lists at googlemail.com Sun Dec 22 16:39:48 2019 From: mirkok.lists at googlemail.com (Mirko) Date: Sun, 22 Dec 2019 22:39:48 +0100 Subject: Most elegant way to do something N times In-Reply-To: References: Message-ID: <5DFFE2A4.5010508@googlemail.com> Am 22.12.2019 um 21:34 schrieb Batuhan Taskaya: > I encounter with cases like doing a function 6 time with no argument, or > same arguments over and over or doing some structral thing N times and I > dont know how elegant I can express that to the code. I dont know why but I > dont like this for _ in range(n): do() thing. Any suggestions? > Looks like perfectly legitimate code to me and 'for' is meant to do exactly this, execute something N times. However, you can always to something like this: In [28]: def f(a, b, c): ...: print(a, b, c) ...: In [29]: def repeat(func, times, *args): ...: for _ in range(times): func(*args) ...: In [30]: repeat(f, 6, 1, 2, 3) (1, 2, 3) (1, 2, 3) (1, 2, 3) (1, 2, 3) (1, 2, 3) (1, 2, 3) From python.list at tim.thechases.com Sun Dec 22 16:09:42 2019 From: python.list at tim.thechases.com (Tim Chase) Date: Sun, 22 Dec 2019 15:09:42 -0600 Subject: Most elegant way to do something N times In-Reply-To: References: Message-ID: <20191222150942.01a47791@bigbox.attlocal.net> On 2019-12-22 23:34, Batuhan Taskaya wrote: > I encounter with cases like doing a function 6 time with no > argument, or same arguments over and over or doing some structral > thing N times and I dont know how elegant I can express that to the > code. I dont know why but I dont like this > > for _ in range(n): do() The best way to improve it would be to clarify readibility. Why 6 times instead of 5 or 7? Are they retries? MAX_RETRIES = 6 # defined in RFC-3141592 for _ in range(MAX_RETRIES): if successful(do_thing()): break else: error_condition() or because someone likes the number 6? MY_FAVORITE_NUMBER = 6 for _ in range(MY_FAVORITE_NUMBER): print(f"I love the number {MY_FAVORITE_NUMBER}!") or days of the week that aren't Sunday? for day in calendar.c.iterweekdays(): if day == calendar.SUNDAY: continue do_thing() or is it because it's the number of whole columns that fit? screen_cols = 80 # get this from curses? chars_per_col = 13 for _ in range(screen_cols // chars_per_col): do_thing() or because that's how many people there are in a paticular grouping? family = "Dad Mom Sis Brother Me Baby".split() for _ in family: do_thing() Note how each of those conveys what "6" means, not just some arbitrary number. -tkc From python at python.invalid Sun Dec 22 18:02:02 2019 From: python at python.invalid (Python) Date: Mon, 23 Dec 2019 00:02:02 +0100 Subject: Most elegant way to do something N times In-Reply-To: References: Message-ID: <5dfff5b9$0$5893$426a74cc@news.free.fr> Le 22/12/2019 ? 23:50, Stefan Ram a ?crit?: > Batuhan Taskaya writes: >> dont like this for _ in range(n): do() thing. Any suggestions? > > The participants of my beginner classes like to write: > > n * do() > > for that purpose, but usuallythis does not work as intended! > > What would work sometimes is > > eval( n * 'do();' ) > or: class repeatly_do: def __init__(self,n): self.n = n def __mul__(self,action): for _ in range(self.n): action() def do(): print('Here I am...') repeatly_do(5) * do From hjp-python at hjp.at Mon Dec 23 10:43:18 2019 From: hjp-python at hjp.at (Peter J. Holzer) Date: Mon, 23 Dec 2019 16:43:18 +0100 Subject: Transfer a file to httpserver via POST command from curl In-Reply-To: References: <20191217184404.298104a4@nxp10225> <20191222181838.3918384c@nxp10225> Message-ID: <20191223154318.GA20612@hjp.at> On 2019-12-23 04:24:22 +1100, Chris Angelico wrote: > On Mon, Dec 23, 2019 at 4:20 AM wrote: > > On Wed, 18 Dec 2019 04:52:33 +1100 > > Chris Angelico wrote: > > > > > On Wed, Dec 18, 2019 at 4:45 AM wrote: > > > > BTW, the canonical way to upload files via http is PUT, not POST. > > > > You might want to look into that, but here it is off-topic. > > > > > > Citation needed. > > > > https://tools.ietf.org/html/rfc2616#page-55 > > Yes, that's the definition of PUT - but what does that mean in terms > of file uploads? POST is also valid. Look at its example usages. ACK. Also the first sentence reads: | The PUT method requests that the enclosed entity be stored under the | supplied Request-URI. I understand this to mean that when I send a PUT https://example.com/foo/bar request and receive a 201 or 204 response, a subsequent GET https://example.com/foo/bar will return the body I sent in the PUT request. This is fine if the client can determine the URI of the response. Quite often this not what you want. You want the client to supply the data but let the server determine what to do with it. For example, the server might just number the uploaded files (or use a timestamp or a hash). Or the uploaded file may not be available under any URL at all, but is used to modify other resources and then discarded. In these cases PUT is inappropriate and POST is appropriate. And sometimes the uploaded file is only part of the data (think of HTML forms with a file input field). Then POST is also the correct choice. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp at hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From rosuav at gmail.com Mon Dec 23 12:08:33 2019 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 24 Dec 2019 04:08:33 +1100 Subject: Transfer a file to httpserver via POST command from curl In-Reply-To: <20191223154318.GA20612@hjp.at> References: <20191217184404.298104a4@nxp10225> <20191222181838.3918384c@nxp10225> <20191223154318.GA20612@hjp.at> Message-ID: On Tue, Dec 24, 2019 at 2:44 AM Peter J. Holzer wrote: > > On 2019-12-23 04:24:22 +1100, Chris Angelico wrote: > > On Mon, Dec 23, 2019 at 4:20 AM wrote: > > > On Wed, 18 Dec 2019 04:52:33 +1100 > > > Chris Angelico wrote: > > > > > > > On Wed, Dec 18, 2019 at 4:45 AM wrote: > > > > > BTW, the canonical way to upload files via http is PUT, not POST. > > > > > You might want to look into that, but here it is off-topic. > > > > > > > > Citation needed. > > > > > > https://tools.ietf.org/html/rfc2616#page-55 > > > > Yes, that's the definition of PUT - but what does that mean in terms > > of file uploads? POST is also valid. Look at its example usages. > > ACK. Also the first sentence reads: > > | The PUT method requests that the enclosed entity be stored under the > | supplied Request-URI. > > I understand this to mean that when I send a > > PUT https://example.com/foo/bar > > request and receive a 201 or 204 response, a subsequent > > GET https://example.com/foo/bar > > will return the body I sent in the PUT request. > > This is fine if the client can determine the URI of the response. > > Quite often this not what you want. You want the client to supply the > data but let the server determine what to do with it. For example, the > server might just number the uploaded files (or use a timestamp or a > hash). Or the uploaded file may not be available under any URL at all, > but is used to modify other resources and then discarded. > > In these cases PUT is inappropriate and POST is appropriate. Your understanding is exactly correct. PUT is appropriate when you already know the URI, POST is appropriate when you want to create a brand new item. The latter is actually far more common. > And sometimes the uploaded file is only part of the data (think of HTML > forms with a file input field). Then POST is also the correct choice. Right, although APIs will often work differently from forms. For instance, the Twitch video uploading protocol starts with a simple POST request with some metadata to create a video, followed by a series of PUT requests to submit each section of the file, and finally a POST to complete the upload. But that only works because all the PUT requests use a predictable URI based on an upload token (created by the initial POST request), which isn't a normal way to build a file upload. (Twitch's video upload can take some pretty ginormous files. You don't want a simple connection glitch to force you to reupload the entire thing from scratch.) But your analysis of POST vs PUT is absolutely correct. File uploads can be done either way, but only use PUT if you know the URI that the object is to live at, which usually means that you can subsequently GET that and retrieve your original file. ChrisA From elbarbun at gmail.com Mon Dec 23 18:12:25 2019 From: elbarbun at gmail.com (Marco Sulla) Date: Tue, 24 Dec 2019 00:12:25 +0100 Subject: Most elegant way to do something N times In-Reply-To: References: Message-ID: > > I encounter with cases like doing a function 6 time with no argument, or > same arguments over and over or doing some structral thing N times and I > dont know how elegant I can express that to the code. > ??? Excuse me, but why you needed to call the same function SIX times? This seems to me not elegant in primis. Can you give us a practical example? On Sun, 22 Dec 2019 at 21:38, Batuhan Taskaya wrote: > I encounter with cases like doing a function 6 time with no argument, or > same arguments over and over or doing some structral thing N times and I > dont know how elegant I can express that to the code. I dont know why but I > dont like this for _ in range(n): do() thing. Any suggestions? > -- > https://mail.python.org/mailman/listinfo/python-list > From rosuav at gmail.com Mon Dec 23 19:04:22 2019 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 24 Dec 2019 11:04:22 +1100 Subject: Most elegant way to do something N times In-Reply-To: References: Message-ID: On Tue, Dec 24, 2019 at 10:45 AM Marco Sulla wrote: > > > > > I encounter with cases like doing a function 6 time with no argument, or > > same arguments over and over or doing some structral thing N times and I > > dont know how elegant I can express that to the code. > > > > ??? Excuse me, but why you needed to call the same function SIX times? This > seems to me not elegant in primis. > > Can you give us a practical example? > File parsing. You read a section header and want to ignore that section, so you ignore the next 15 lines. ChrisA From morphex at gmail.com Mon Dec 23 19:16:49 2019 From: morphex at gmail.com (Morten W. Petersen) Date: Tue, 24 Dec 2019 01:16:49 +0100 Subject: Most elegant way to do something N times In-Reply-To: References: Message-ID: It depends on what's elegant, and beauty is in the eye of the beholder.. def recurse(n): print(?do be do be do?) if n>0: recurse(n-1) Regards, Morten s?n. 22. des. 2019, 21:37 skrev Batuhan Taskaya : > I encounter with cases like doing a function 6 time with no argument, or > same arguments over and over or doing some structral thing N times and I > dont know how elegant I can express that to the code. I dont know why but I > dont like this for _ in range(n): do() thing. Any suggestions? > -- > https://mail.python.org/mailman/listinfo/python-list > From PythonList at DancesWithMice.info Mon Dec 23 19:27:09 2019 From: PythonList at DancesWithMice.info (DL Neil) Date: Tue, 24 Dec 2019 13:27:09 +1300 Subject: Most elegant way to do something N times In-Reply-To: References: Message-ID: <189be71f-477d-4f32-8c97-f989d8a8862c@DancesWithMice.info> On 24/12/19 1:04 PM, Chris Angelico wrote: > On Tue, Dec 24, 2019 at 10:45 AM Marco Sulla wrote: >> >>> >>> I encounter with cases like doing a function 6 time with no argument, or >>> same arguments over and over or doing some structral thing N times and I >>> dont know how elegant I can express that to the code. >>> >> >> ??? Excuse me, but why you needed to call the same function SIX times? This >> seems to me not elegant in primis. >> >> Can you give us a practical example? Taking the top/bottom six from a sorted list of occurrences. eg in Accounts Payable, a clerk might be given the task of 'chasing' the six largest debtors (people who owe us money), as today's work assignment/priority. > File parsing. You read a section header and want to ignore that > section, so you ignore the next 15 lines. (just to be cheeky to @Chris) Perhaps better as a Finite State Machine, with one state being 'seeking section header'. 1 can we guarantee that the 'magic constant' of 15 will always apply? 2 presumably the total routine will involve more than identifying a single header and skipping (only) that section. -- Regards =dn From rosuav at gmail.com Mon Dec 23 19:41:51 2019 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 24 Dec 2019 11:41:51 +1100 Subject: Most elegant way to do something N times In-Reply-To: <189be71f-477d-4f32-8c97-f989d8a8862c@DancesWithMice.info> References: <189be71f-477d-4f32-8c97-f989d8a8862c@DancesWithMice.info> Message-ID: On Tue, Dec 24, 2019 at 11:34 AM DL Neil via Python-list wrote: > > On 24/12/19 1:04 PM, Chris Angelico wrote: > > File parsing. You read a section header and want to ignore that > > section, so you ignore the next 15 lines. > > (just to be cheeky to @Chris) > > Perhaps better as a Finite State Machine, with one state being 'seeking > section header'. > 1 can we guarantee that the 'magic constant' of 15 will always apply? That number came from the header itself. Okay, so more commonly this kind of thing is a *byte* count, but I have known some that count lines, or entries, or something else that you'd have a function to grab one of. > 2 presumably the total routine will involve more than identifying a > single header and skipping (only) that section. Of course. But after skipping that section (by ignoring X lines), you go do something else. ChrisA From info at tundraware.com Mon Dec 23 19:48:52 2019 From: info at tundraware.com (Tim Daneliuk) Date: Mon, 23 Dec 2019 18:48:52 -0600 Subject: Lists And Missing Commas Message-ID: If I do this: foo = [ "bar", "baz" "slop", "crud" ] Python silently accepts that and makes the middle term "bazslop". BUT, if I do this: foo = [ "bar", "baz" 1, "crud" ] or this: foo = [ "bar", 2 1, "crud" ] The interpreter throws a syntax error. This is more of an intellectual curiosity than anything else, but why do strings silently concatenate like that, while all other case blow up with an error. i.e., What is about the language the promotes this behavior. At first blush, it seems inconsistent, but what do I know ... From aaronngray.lists at gmail.com Mon Dec 23 20:01:22 2019 From: aaronngray.lists at gmail.com (Aaron Gray) Date: Tue, 24 Dec 2019 01:01:22 +0000 Subject: Lists And Missing Commas In-Reply-To: References: Message-ID: On Tuesday, 24 December 2019, Tim Daneliuk wrote: > If I do this: > > foo = [ "bar", "baz" "slop", "crud" ] > > Python silently accepts that and makes the middle term "bazslop". Strings concatinate over line endings so this case is only sensible really. > > BUT, if I do this: > > foo = [ "bar", "baz" 1, "crud" ] > > or this: > > foo = [ "bar", 2 1, "crud" ] > > The interpreter throws a syntax error. > > This is more of an intellectual curiosity than anything else, but why do > strings silently > concatenate like that, while all other case blow up with an error. i.e., > What is about > the language the promotes this behavior. At first blush, it seems > inconsistent, but what > do I know ... > > > > > -- > https://mail.python.org/mailman/listinfo/python-list > -- Independent Software Engineer and amateur Computer Scientist From avigross at verizon.net Mon Dec 23 20:51:24 2019 From: avigross at verizon.net (Avi Gross) Date: Mon, 23 Dec 2019 20:51:24 -0500 Subject: Most elegant way to do something N times In-Reply-To: References: Message-ID: <006701d5b9fc$a6183880$f248a980$@verizon.net> I would like some examples of how one does what is requested in some other programming languages. I doubt there is much need of a shorter way to do anything N times and throw away any return values. Python has many ways to do just about anything. It has some features which suggest a particular way that many accept as pythonic. If there was a need to do something like: repeat_n( times=6, funct=fun, moreargs=whatever) then someone probably would have included it. So, yes, a fairly short and often clear method is: for _ in range(6): ... And others have shown multiple ways to get something similar to what you want. Want real clarity for exactly six times? # Do this six times: whatever() whatever() whatever() whatever() whatever() whatever() Heck, the above may even run faster without loop overhead. Like Marco, it would be interesting to hear actual examples where it makes sense to repeat something the same way N times. Many such examples do have ways to do it trivially. I mean if you want to line up text and need N copies of say an asterisk or space, you can often use something like "*" * 5 And the techniques to repeat some function call N times with the arguments varying each time, can be handled many other ways too including map functions and comprehensions. I am left wondering if this is not about the real world but simply an academic exercise. -----Original Message----- From: Python-list On Behalf Of Marco Sulla Sent: Monday, December 23, 2019 6:12 PM To: Batuhan Taskaya Cc: Ian Kelly ian.g.kelly-at-gmail.com |python-list at python.org| Subject: Re: Most elegant way to do something N times > > I encounter with cases like doing a function 6 time with no argument, > or same arguments over and over or doing some structral thing N times > and I dont know how elegant I can express that to the code. > ??? Excuse me, but why you needed to call the same function SIX times? This seems to me not elegant in primis. Can you give us a practical example? On Sun, 22 Dec 2019 at 21:38, Batuhan Taskaya wrote: > I encounter with cases like doing a function 6 time with no argument, > or same arguments over and over or doing some structral thing N times > and I dont know how elegant I can express that to the code. I dont > know why but I dont like this for _ in range(n): do() thing. Any suggestions? > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list From PythonList at DancesWithMice.info Mon Dec 23 20:52:41 2019 From: PythonList at DancesWithMice.info (DL Neil) Date: Tue, 24 Dec 2019 14:52:41 +1300 Subject: Lists And Missing Commas In-Reply-To: References: Message-ID: On 24/12/19 1:48 PM, Tim Daneliuk wrote: > If I do this: > > foo = [ "bar", "baz" "slop", "crud" ] > > Python silently accepts that and makes the middle term "bazslop". > > BUT, if I do this: > > foo = [ "bar", "baz" 1, "crud" ] > > or this: > > foo = [ "bar", 2 1, "crud" ] > > The interpreter throws a syntax error. > > This is more of an intellectual curiosity than anything else, but why do strings silently > concatenate like that, while all other case blow up with an error. i.e., What is about > the language the promotes this behavior. At first blush, it seems inconsistent, but what > do I know ... Restricting our conversation to numbers/integers and strings, "concatenate" only ever seems to apply to string objects. (there are meanings of "concatenate" for matrices, for example) For fun, try: help( str ) # and help( int ) They both mention __add__(self, value, /) In both cases such is defined as "self+value" - but remember that the "+" operator is "over-loaded". Thus (and I'm telling you nothing new here) when dealing with strings we read "+" as "concatenate"; whereas with integers it is plainly "add". Two quite different operators/operations! However, your point involves the fact that whereas: 1 + 2 # 3 is *clearly* addition, and "a" + "b" # "ab" is *clearly* concatenation "a" "b" # also evaluates to "ab" and is thus, concatenation without any explicit infix operator! Just one cotton-picking minute - what's going on here? The answer to all your dreams (um, maybe...) lies in "The Python Language Reference" docs, particularly Section 2 "Lexical Analysis" (the process of taking "tokens" and making sense of how they go-together!). <<< 2.4.2. String literal concatenation Multiple adjacent string or bytes literals (delimited by whitespace), possibly using different quoting conventions, are allowed, and their meaning is the same as their concatenation. Thus, "hello" 'world' is equivalent to "helloworld". This feature can be used to reduce the number of backslashes needed, to split long strings conveniently across long lines, or even to add comments to parts of strings, for example: re.compile("[A-Za-z_]" # letter or underscore "[A-Za-z0-9_]*" # letter, digit or underscore ) Note that this feature is defined at the syntactical level, but implemented at compile time. The ?+? operator must be used to concatenate string expressions at run time. Also note that literal concatenation can use different quoting styles for each component (even mixing raw strings and triple quoted strings), and formatted string literals may be concatenated with plain string literals. >>> Thus, the adjacency of two literals explicitly only implies concatenation for strings. There is no equivalent/similar mention for numbers. WebRef: https://docs.python.org/3/reference/lexical_analysis.html -- Regards =dn From avigross at verizon.net Mon Dec 23 21:17:17 2019 From: avigross at verizon.net (Avi Gross) Date: Mon, 23 Dec 2019 21:17:17 -0500 Subject: Most elegant way to do something N times In-Reply-To: <189be71f-477d-4f32-8c97-f989d8a8862c@DancesWithMice.info> References: <189be71f-477d-4f32-8c97-f989d8a8862c@DancesWithMice.info> Message-ID: <006801d5ba00$44179460$cc46bd20$@verizon.net> What others have answered is tangential. Nobody doubts there are places where you want to repeat something N times. Just to add an example, if you are parsing something nested by parentheses or perhaps HTML, and you reach a point where you have seen N opening symbols and start seeing a closing symbols, you might want to swallow the next N-1 such symbols before moving on to whatever comes next. Yes, I know that can be more complex. But in many such cases, people do not just count lines or whatever, but actively examine each line. This is especially true where it is valid to have say a comment embedded or other things that make the count uncertain. Again, what language does something simpler? Example, in R, is this much better? for(n_times in 1:6) {print("hi")} [1] "hi" [1] "hi" [1] "hi" [1] "hi" [1] "hi" [1] "hi" Using R, again, you can make a repeat function that calls your designated function with a fizxed argument N times: repetito = function(n, func, arg) { for (i in 1:n) { func(arg) } } repetito(6, print, "hi") [1] "hi" [1] "hi" [1] "hi" [1] "hi" [1] "hi" [1] "hi" Python allows similar ways: >>> def repetito(n, func, args) : for _ in range(n): func(args) >>> repetito(6, print, "hi") hi hi hi hi hi hi You can modify these ideas to having no arguments or multiple arguments or even varying numbers of arguments with varying numbers of positional and keyword arguments. You need to be careful at times to make sure that the calls are not evaluated once but each time. I suspect the request boils down to wanting yet another keyword or two added to a language that does not need more bloat. -----Original Message----- From: Python-list On Behalf Of DL Neil via Python-list Sent: Monday, December 23, 2019 7:27 PM To: Python Subject: Re: Most elegant way to do something N times On 24/12/19 1:04 PM, Chris Angelico wrote: > On Tue, Dec 24, 2019 at 10:45 AM Marco Sulla wrote: >> >>> >>> I encounter with cases like doing a function 6 time with no >>> argument, or same arguments over and over or doing some structral >>> thing N times and I dont know how elegant I can express that to the code. >>> >> >> ??? Excuse me, but why you needed to call the same function SIX >> times? This seems to me not elegant in primis. >> >> Can you give us a practical example? Taking the top/bottom six from a sorted list of occurrences. eg in Accounts Payable, a clerk might be given the task of 'chasing' the six largest debtors (people who owe us money), as today's work assignment/priority. > File parsing. You read a section header and want to ignore that > section, so you ignore the next 15 lines. (just to be cheeky to @Chris) Perhaps better as a Finite State Machine, with one state being 'seeking section header'. 1 can we guarantee that the 'magic constant' of 15 will always apply? 2 presumably the total routine will involve more than identifying a single header and skipping (only) that section. -- Regards =dn -- https://mail.python.org/mailman/listinfo/python-list From rosuav at gmail.com Mon Dec 23 21:35:04 2019 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 24 Dec 2019 13:35:04 +1100 Subject: Lists And Missing Commas In-Reply-To: References: Message-ID: On Tue, Dec 24, 2019 at 12:56 PM DL Neil via Python-list wrote: > However, your point involves the fact that whereas: > > 1 + 2 # 3 is *clearly* addition, and > "a" + "b" # "ab" is *clearly* concatenation > > "a" "b" # also evaluates to "ab" > > and is thus, concatenation without any explicit infix operator! Just one > cotton-picking minute - what's going on here? 1_2 # Clearly concatenation. Right? 0_0 # Clearly an emoticon Just another cotton-picking minute...... ChrisA From PythonList at DancesWithMice.info Mon Dec 23 22:34:47 2019 From: PythonList at DancesWithMice.info (DL Neil) Date: Tue, 24 Dec 2019 16:34:47 +1300 Subject: Lists And Missing Commas In-Reply-To: References: Message-ID: <49476692-6aa9-e017-69d5-6c53cb89fc46@DancesWithMice.info> On 24/12/19 3:35 PM, Chris Angelico wrote: > On Tue, Dec 24, 2019 at 12:56 PM DL Neil via Python-list > wrote: >> However, your point involves the fact that whereas: >> >> 1 + 2 # 3 is *clearly* addition, and >> "a" + "b" # "ab" is *clearly* concatenation >> >> "a" "b" # also evaluates to "ab" >> >> and is thus, concatenation without any explicit infix operator! Just one >> cotton-picking minute - what's going on here? > > 1_2 # Clearly concatenation. Right? > 0_0 # Clearly an emoticon > > Just another cotton-picking minute...... Shows what you know: 1_2 is marching orders - and here are yours... -- Regards =dn From info at tundraware.com Mon Dec 23 23:20:53 2019 From: info at tundraware.com (Tim Daneliuk) Date: Mon, 23 Dec 2019 22:20:53 -0600 Subject: Lists And Missing Commas In-Reply-To: References: Message-ID: <5ag8dg-lcn2.ln1@oceanview.tundraware.com> On 12/23/19 7:52 PM, DL Neil wrote: > > WebRef:?https://docs.python.org/3/reference/lexical_analysis.html Yep, that explains it, but it still feels non-regular to me. From a pointy headed academic POV, I'd like to see behavior consistent across types. Again ... what do I know? From info at tundraware.com Mon Dec 23 23:21:40 2019 From: info at tundraware.com (Tim Daneliuk) Date: Mon, 23 Dec 2019 22:21:40 -0600 Subject: Lists And Missing Commas In-Reply-To: References: Message-ID: On 12/23/19 8:35 PM, Chris Angelico wrote: > On Tue, Dec 24, 2019 at 12:56 PM DL Neil via Python-list > wrote: >> However, your point involves the fact that whereas: >> >> 1 + 2 # 3 is *clearly* addition, and >> "a" + "b" # "ab" is *clearly* concatenation >> >> "a" "b" # also evaluates to "ab" >> >> and is thus, concatenation without any explicit infix operator! Just one >> cotton-picking minute - what's going on here? > > 1_2 # Clearly concatenation. Right? > 0_0 # Clearly an emoticon > > Just another cotton-picking minute...... > > ChrisA > You are excused and required to do 30 days penance writing .NET. From frank at chagford.com Tue Dec 24 00:46:00 2019 From: frank at chagford.com (Frank Millman) Date: Tue, 24 Dec 2019 07:46:00 +0200 Subject: Lists And Missing Commas In-Reply-To: <5ag8dg-lcn2.ln1@oceanview.tundraware.com> References: <5ag8dg-lcn2.ln1@oceanview.tundraware.com> Message-ID: On 2019-12-24 6:20 AM, Tim Daneliuk wrote: > On 12/23/19 7:52 PM, DL Neil wrote: >> >> WebRef:?https://docs.python.org/3/reference/lexical_analysis.html > > > Yep, that explains it, but it still feels non-regular to me. From a pointy headed academic > POV, I'd like to see behavior consistent across types. Again ... what do I know? > From the Zen, 'Practicality beats purity'. From the docs - """ String literals that are part of a single expression and have only whitespace between them will be implicitly converted to a single string literal. That is, ("spam " "eggs") == "spam eggs". """ I do not see it as 'concatenation', rather as a way of constructing a single string from a number of smaller chunks. The docs talk about 'whitespace', but I would guess that the use of a single space is uncommon. More likely is the use of a newline. I use this from time to time when constructing long string literals - long_string = ( "this is the first chunk " "this is the second chunk " "etc etc" ) My 0.02c Frank Millman From PythonList at DancesWithMice.info Tue Dec 24 00:46:34 2019 From: PythonList at DancesWithMice.info (DL Neil) Date: Tue, 24 Dec 2019 18:46:34 +1300 Subject: Lists And Missing Commas In-Reply-To: <5ag8dg-lcn2.ln1@oceanview.tundraware.com> References: <5ag8dg-lcn2.ln1@oceanview.tundraware.com> Message-ID: <97d9350c-3f4a-1143-a5fb-ffaec2661c4c@DancesWithMice.info> On 24/12/19 5:20 PM, Tim Daneliuk wrote: > On 12/23/19 7:52 PM, DL Neil wrote: >> >> WebRef:?https://docs.python.org/3/reference/lexical_analysis.html > > > Yep, that explains it, but it still feels non-regular to me. From a pointy headed academic > POV, I'd like to see behavior consistent across types. Again ... what do I know? I thought it was 'the boss' who has the "pointy head'? (so that any 'difficult' idea will not land, but simply slide-off to one side or the other?) It is an interesting lateral-thinking enquiry, and we should not simply accept 'stuff', but question how it best works/we can best use it. That said, on the surface, I might agree. Venturing into compiler-writing/lexical analysis, and comparing with other languages, we rapidly realise that it is 'odd' (or special). However, keep reading (the above web.ref) and find the section where "white space" is described. Combine that with the idea/nuisance-value of splitting long strings over multiple lines. Rather than puzzling-over an 'only/special/weird concatenation' allowance, perhaps we should see a 'convenience factor'? -- Regards =dn From frank at chagford.com Tue Dec 24 00:46:00 2019 From: frank at chagford.com (Frank Millman) Date: Tue, 24 Dec 2019 07:46:00 +0200 Subject: Lists And Missing Commas In-Reply-To: <5ag8dg-lcn2.ln1@oceanview.tundraware.com> References: <5ag8dg-lcn2.ln1@oceanview.tundraware.com> Message-ID: On 2019-12-24 6:20 AM, Tim Daneliuk wrote: > On 12/23/19 7:52 PM, DL Neil wrote: >> >> WebRef:?https://docs.python.org/3/reference/lexical_analysis.html > > > Yep, that explains it, but it still feels non-regular to me. From a pointy headed academic > POV, I'd like to see behavior consistent across types. Again ... what do I know? > From the Zen, 'Practicality beats purity'. From the docs - """ String literals that are part of a single expression and have only whitespace between them will be implicitly converted to a single string literal. That is, ("spam " "eggs") == "spam eggs". """ I do not see it as 'concatenation', rather as a way of constructing a single string from a number of smaller chunks. The docs talk about 'whitespace', but I would guess that the use of a single space is uncommon. More likely is the use of a newline. I use this from time to time when constructing long string literals - long_string = ( "this is the first chunk " "this is the second chunk " "etc etc" ) My 0.02c Frank Millman From info at tundraware.com Tue Dec 24 10:45:42 2019 From: info at tundraware.com (Tim Daneliuk) Date: Tue, 24 Dec 2019 09:45:42 -0600 Subject: Lists And Missing Commas In-Reply-To: References: <5ag8dg-lcn2.ln1@oceanview.tundraware.com> Message-ID: <6eo9dg-jve2.ln1@oceanview.tundraware.com> On 12/24/19 6:37 AM, Stefan Ram wrote: > And you all are aware that this kind of string concatenation > happens in C and C++, too, aren't you? > > main.c > > #include > int main( void ){ puts( "a" "b" ); } > > transcript > > ab Noting that it has been a long time since I looked at the C specification ... Is the above an artifact of how puts() is implemented or is it innate in the language spec? From Richard at Damon-Family.org Tue Dec 24 11:14:25 2019 From: Richard at Damon-Family.org (Richard Damon) Date: Tue, 24 Dec 2019 11:14:25 -0500 Subject: Lists And Missing Commas In-Reply-To: <6eo9dg-jve2.ln1@oceanview.tundraware.com> References: <5ag8dg-lcn2.ln1@oceanview.tundraware.com> <6eo9dg-jve2.ln1@oceanview.tundraware.com> Message-ID: <440f9741-6fc5-80d0-0d61-9a275d3365a4@Damon-Family.org> On 12/24/19 10:45 AM, Tim Daneliuk wrote: > On 12/24/19 6:37 AM, Stefan Ram wrote: >> And you all are aware that this kind of string concatenation >> happens in C and C++, too, aren't you? >> >> main.c >> >> #include >> int main( void ){ puts( "a" "b" ); } >> >> transcript >> >> ab > Noting that it has been a long time since I looked at the C specification ... > > Is the above an artifact of how puts() is implemented or is it innate in the language spec? The automatic concatenation of adjacent strings is an innate part of the C language spec. One important reason this was included in the C language was so that some of the predefined macros with a string value (like the compilation time __TIME__) could be built into a string constant. A second is that C had no equivalent to """ as a multi-line string constant, and using line continuation was problematical on some systems using fixed length lines, so allowing a constant to be built on multiple lines was useful. -- Richard Damon From guido at python.org Tue Dec 24 11:20:05 2019 From: guido at python.org (Guido van Rossum) Date: Tue, 24 Dec 2019 09:20:05 -0700 Subject: [ANN] Retrospective of Python compilation efforts In-Reply-To: <20191224115535.565697cc@x230> References: <20191224115535.565697cc@x230> Message-ID: Nice work. On Tue, Dec 24, 2019 at 2:59 AM Paul Sokolovsky wrote: > Hello, > > Over the years, the Python community produced many compiler projects for > the language, second to probably only C and LISP. Majority are of > course of research and proof of concept quality, but there're a number > of quite advanced projects. > > For some time, I was trying to compile a retrospective of these > efforts, formatted as a github "awesome" list: > > https://github.com/pfalcon/awesome-python-compilers > > > Happy holidays, > Paul > -- > Python-announce-list mailing list -- python-announce-list at python.org > To unsubscribe send an email to python-announce-list-leave at python.org > https://mail.python.org/mailman3/lists/python-announce-list.python.org/ > > Support the Python Software Foundation: > http://www.python.org/psf/donations/ > -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* From avigross at verizon.net Tue Dec 24 13:02:03 2019 From: avigross at verizon.net (Avi Gross) Date: Tue, 24 Dec 2019 13:02:03 -0500 Subject: Lists And Missing Commas In-Reply-To: References: Message-ID: <007601d5ba84$3f270970$bd751c50$@verizon.net> As a purist, it makes people uncomfortable if all 'objects' are not treated alike. But I look at the question from a definition and parsing mechanism view. When an interpreter (or compiler) reads a program, it often does it in phases and tries to tokenize parts. So, the definition of something often makes it look ahead or back or even sideways to see how much of the program text makes up a single entity. The definition of a NUMBER is complex but clearly it needs to allow underscores or designations that make it octal versus decimal or a floating point number or even scientific notation. The point is it does some looking ahead to make sure it gets the entire number and then sets it aside as a token for a single entity. The definition of a fixed string is also complex as it can be a binary string or other formats but the focus here is that string literals were DESIGNED to be able to span multiple lines or be put in piecewise. So the parser does not stop when it sees "Hello " but continues to see if there is more that can be used to make a full string literal. It has to skip past whitespace and see what follows. If what follows is "World!" then it makes "Hello World!" and keeps scanning beyond more whitespace until something is found that cannot be considered part of the same literal. Other languages use this technique such as C++ An analogy might be how a compiler in a language that supports constants will look at a line declaring something like this: const int volume = 5 * 6 * 7 Would it not be possible that the compiler, knowing this is a constant, would not pre-calculate the result and act as if you had written: const int volume = 210 The run time, would not know or care. So, if the analogy makes sense, in python the early evaluation phase is required to combine as much as possible into a single string literal. It is a bit like a regular expression that matches a pattern greedily unless asked not to. This tokenization probably precedes the point where it sees a series of commas in a context such as a list definition or function call argument list. Again, a purist might want all objects, within reason, to look alike. There are other mechanisms to gain this concatenation functionality in strings. This being Python (which lies about how there should be one unique way to logically do something) there may be dozens of ways to get this functionality starting with using a PLUS operator and continuing with one of 5 or so ways to build a string from other objects as well as strategic uses of the backslash character and so on. But it is a feature that is there and seems to also be in some other languages. It is generally easy to avoid using this implicit concatenation but I think an important point is that you may inadvertently get the wrong result when accidentally leaving out a comma. That is true. I note this is an old known issue: https://www.python.org/dev/peps/pep-3126/ There are some lint programs that check your code and supply warnings and I see some languages have the option to generate warnings when the two strings are on the same line. I wonder if a Python lint does that. It may at least warn of this usage in time to check the code and put back the comma. -----Original Message----- From: Python-list On Behalf Of Tim Daneliuk Sent: Monday, December 23, 2019 11:22 PM To: python-list at python.org Subject: Re: Lists And Missing Commas On 12/23/19 8:35 PM, Chris Angelico wrote: > On Tue, Dec 24, 2019 at 12:56 PM DL Neil via Python-list > wrote: >> However, your point involves the fact that whereas: >> >> 1 + 2 # 3 is *clearly* addition, and >> "a" + "b" # "ab" is *clearly* concatenation >> >> "a" "b" # also evaluates to "ab" >> >> and is thus, concatenation without any explicit infix operator! Just >> one cotton-picking minute - what's going on here? > > 1_2 # Clearly concatenation. Right? > 0_0 # Clearly an emoticon > > Just another cotton-picking minute...... > > ChrisA > You are excused and required to do 30 days penance writing .NET. -- https://mail.python.org/mailman/listinfo/python-list From ethan at stoneleaf.us Tue Dec 24 13:26:27 2019 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 24 Dec 2019 10:26:27 -0800 Subject: Lists And Missing Commas In-Reply-To: <007601d5ba84$3f270970$bd751c50$@verizon.net> References: <007601d5ba84$3f270970$bd751c50$@verizon.net> Message-ID: On 12/24/2019 10:02 AM, Avi Gross via Python-list wrote: > This being Python (which lies about how there should be one unique way to > logically do something) The koan is: There should be one-- and preferably only one --obvious way to do it. It is not: - only one way - one unique way - the one true way - the way to end all ways ;) - etc It is also `preferable`, not: - absolute - eternal - fixed - exclusive -- ~Ethan~ From mail.python.org at marco.sulla.e4ward.com Tue Dec 24 15:41:49 2019 From: mail.python.org at marco.sulla.e4ward.com (Marco Sulla) Date: Tue, 24 Dec 2019 21:41:49 +0100 Subject: Most elegant way to do something N times In-Reply-To: References: Message-ID: On Tue, 24 Dec 2019 at 01:07, Chris Angelico wrote: > On Tue, Dec 24, 2019 at 10:45 AM Marco Sulla <...> wrote: > > ??? Excuse me, but why you needed to call the same function SIX times? This > > seems to me not elegant in primis. > > > > Can you give us a practical example? > > File parsing. You read a section header and want to ignore that > section, so you ignore the next 15 lines. mmap and find? On Tue, 24 Dec 2019 at 01:35, DL Neil via Python-list wrote: > Taking the top/bottom six from a sorted list of occurrences. Slicing? From avigross at verizon.net Tue Dec 24 16:48:26 2019 From: avigross at verizon.net (Avi Gross) Date: Tue, 24 Dec 2019 16:48:26 -0500 Subject: Lists And Extra Commas at end References: <00ba01d5baa3$df8db7a0$9ea926e0$.ref@verizon.net> Message-ID: <00ba01d5baa3$df8db7a0$9ea926e0$@verizon.net> Marco, Python is used by some in an interactive mode but also in more of a batch mode such as on a server. The former can possibly see a SyntaxWarning. Do you want that as a default or something you set when you start the Python Interpreter or perhaps a command within it? I note a brief search to see about Warnings in Jupyter keeps talking about how to turn warnings off! My point is that warnings about what is not a real problem tend to make people ignore all warnings. Let me switch gears to the terminal comma situation. Unlike many languages, Python decided a dangling comma is perfectly allowable in many situations, perhaps all. >>> a=[1,2,3,] >>> a [1, 2, 3] The above does not work for an empty comma at the beginning or middle or a double comma at the end. It is clear why they allowed this as it makes some things easier like building up a list in a loop and not needing to figure out the special case when you are done and suppressing the last comma. Since a function argument list is just a list in Python, the following works fine: def testink(a,b,): pass And, of course, you can use the same dangling comma in making a tuple, dictionary or set and who knows where else. So, is that a feature you want warnings about? After all, a dangling comma may simply mean you left something out and meant to add later? As I see it, languages have tradeoffs and nobody will be completely satisfied so you choose and hope. But, warnings you can turn on an off do indeed make sense. But at what level? If the main interpreter has to do all kinds of testing even for a VALID use of the language so it can warn, that is extra overhead each time it is run. I would prefer something less intrusive. Some languages have a way of saying "use strict" (such as JavaScript or Perl) or a "use warnings" that tells to interpreter to warn about things. Since Python did not go this route, with perhaps a tad of that regarding the __future__ setting, I figured a good place to do this is in a Lint type of program that can be set to look for things that are not so much errors as places to check to make sure things are OK. You would not run that every time, just when the code has been changed. And, if you look for a list of places people make mistakes that are often maddening to trace, there are so many. Would you like a warning that multiple NAMES are almost the same except for capitalization or that you have long_variable_1_of_2 as well as long_variable_2_of_2 which only differ by a tad deep in the middle? At some point, you have to take the training wheels off. Not clear what that point is, sometimes, so I try to only drive vehicles with 4 or six tires, LOL! -----Original Message----- From: forwardedby at e4ward.com On Behalf Of Marco Sulla Sent: Tuesday, December 24, 2019 4:03 PM To: Avi Gross Cc: Python-list.python.org-mail.python.org.marco.sulla.e4ward.com at jq38efsu937.reply.e4ward.com Subject: Re: Lists And Missing Commas On Tue, 24 Dec 2019 at 19:05, Avi Gross via Python-list wrote: > There are some lint programs that check your code and supply warnings > and I see some languages have the option to generate warnings when the > two strings are on the same line. I wonder if a Python lint does that. > It may at least warn of this usage in time to check the code and put back the comma. IMHO it's not something that should be delegated to linter, but to the AST compiler itself. A SyntaxWarning should be displayed. One time I spent HOURS because I forgot one single comma in a list of strings. After that, my life is changed. I'm no more an ingenuous, little boy, that trusted the stacktrace. From cs at cskk.id.au Tue Dec 24 17:12:28 2019 From: cs at cskk.id.au (Cameron Simpson) Date: Wed, 25 Dec 2019 09:12:28 +1100 Subject: Lists And Extra Commas at end In-Reply-To: <00ba01d5baa3$df8db7a0$9ea926e0$@verizon.net> References: <00ba01d5baa3$df8db7a0$9ea926e0$@verizon.net> Message-ID: <20191224221228.GA23410@cskk.homeip.net> On 24Dec2019 16:48, Avi Gross wrote: >Let me switch gears to the terminal comma situation. Unlike many >languages, Python decided a dangling comma is perfectly allowable in >many situations, perhaps all. > >>>> a=[1,2,3,] >>>> a >[1, 2, 3] [...] >And, of course, you can use the same dangling comma in making a tuple, >dictionary or set and who knows where else. >So, is that a feature you want warnings about? After all, a dangling >comma may simply mean you left something out and meant to add later? To my mind the killer argument for trailing commas is things like multiline lists, dicts or parameters. Example: def function( arg1=None, arg2=FOO, ): Imagine this for a comewhat extended set of parameters. Or similar for a big list or dict (eg a "table" as part of a class definition). By always including the trailing comma it is easier to insert or delete lines, and it reduces diff noise if you're using revision control (no commas flickering on and off in the diff output). Cheers, Cameron Simpson From mail.python.org at marco.sulla.e4ward.com Tue Dec 24 17:35:40 2019 From: mail.python.org at marco.sulla.e4ward.com (Marco Sulla) Date: Tue, 24 Dec 2019 23:35:40 +0100 Subject: Lists And Extra Commas at end In-Reply-To: <00ba01d5baa3$df8db7a0$9ea926e0$@verizon.net> References: <00ba01d5baa3$df8db7a0$9ea926e0$.ref@verizon.net> <00ba01d5baa3$df8db7a0$9ea926e0$@verizon.net> Message-ID: On Tue, 24 Dec 2019 at 22:51, Avi Gross via Python-list wrote: > So, is that a feature you want warnings about? After all, a dangling comma may simply mean you left something out and meant to add later? .....completely OT. I responded to a topic named "List and missing commas", and suggested a warning for implicit concatenation between adjacent strings. I don't really know why you renamed it "Lists And Extra Commas at end". Have you read the thread and my post? Cheers. From mail.python.org at marco.sulla.e4ward.com Tue Dec 24 16:02:44 2019 From: mail.python.org at marco.sulla.e4ward.com (Marco Sulla) Date: Tue, 24 Dec 2019 22:02:44 +0100 Subject: Lists And Missing Commas In-Reply-To: <007601d5ba84$3f270970$bd751c50$@verizon.net> References: <007601d5ba84$3f270970$bd751c50$@verizon.net> Message-ID: On Tue, 24 Dec 2019 at 19:05, Avi Gross via Python-list wrote: > There are some lint programs that check your code and supply warnings and I > see some languages have the option to generate warnings when the two strings > are on the same line. I wonder if a Python lint does that. It may at least > warn of this usage in time to check the code and put back the comma. IMHO it's not something that should be delegated to linter, but to the AST compiler itself. A SyntaxWarning should be displayed. One time I spent HOURS because I forgot one single comma in a list of strings. After that, my life is changed. I'm no more an ingenuous, little boy, that trusted the stacktrace. From mail.python.org at marco.sulla.e4ward.com Tue Dec 24 18:41:56 2019 From: mail.python.org at marco.sulla.e4ward.com (Marco Sulla) Date: Wed, 25 Dec 2019 00:41:56 +0100 Subject: More efficient/elegant branching In-Reply-To: <9823ad20-6c92-6606-6268-845ec144c5cf@DancesWithMice.info> References: <4c0405e8fe1ca0af482048c60e9ee2f4@posteo.de> <9823ad20-6c92-6606-6268-845ec144c5cf@DancesWithMice.info> Message-ID: I agree with Chris Angelico, branch1 is "the way to go". Maybe you have to add a default at start, maybe None, and maybe raise an exception if res is None. Anyway, despite I'm a pain in the... arse and I usually activate ALL the possible warnings in the world, I always disable cyclomatic complexity warnings in all linters, in any language :D # Off topic - START I proposed a switch-case some years ago. I was brutally condemned like an heretic :D It was something like: when expr match arg1, arg2, ....: [...] match *args: [...] [...] else: # optional [...] expr was evaluated, and checked if equal to one of the arguments. At the first positive check, the relative code was evaluated, and the when breaks. No break keyword, like if-elif. If no check passed and an else clause is present, its code will be evaluated. Anyway this is not helpful in your case, since you have more complicated checks. Mine was only a suggestion to introduce the simple C switch-case in Python, for making easier trivial if-elif chains and for code optimization (C switch-case is optimized by C compilers) # Of topic - END From avigross at verizon.net Tue Dec 24 18:48:48 2019 From: avigross at verizon.net (Avi Gross) Date: Tue, 24 Dec 2019 18:48:48 -0500 Subject: Lists And Extra Commas at end In-Reply-To: <20191224221228.GA23410@cskk.homeip.net> References: <00ba01d5baa3$df8db7a0$9ea926e0$@verizon.net> <20191224221228.GA23410@cskk.homeip.net> Message-ID: <00da01d5bab4$b011bdd0$10353970$@verizon.net> Cameron, I am not at all against the feature. I like it as my programming style is like you describe. One entry per line indented at the same level, in multiple languages. I often do graphics where I generate an image then fine-tune additional parameters to get the effect I want. Some functions take literally hundreds of options to adjust anything from text labels to line types to colors and the high/low values on axes and so on. In languages like R, I tend to put in a final entry with no trailing comma that is something harmless that can be left in position and the same for the first entry. Then my adding/deleting/editing of fields happens in the middle where I always have a terminal comma. My point is the convenience comes with a price for people who make a mistake and are not told perhaps the dangling comma was a placeholder for something to add that they forgot or ... Again, the convenience is in some sense removing a mathematical symmetry, but so what? -----Original Message----- From: Cameron Simpson Sent: Tuesday, December 24, 2019 5:12 PM To: Avi Gross Cc: python-list at python.org Subject: Re: Lists And Extra Commas at end On 24Dec2019 16:48, Avi Gross wrote: >Let me switch gears to the terminal comma situation. Unlike many >languages, Python decided a dangling comma is perfectly allowable in >many situations, perhaps all. > >>>> a=[1,2,3,] >>>> a >[1, 2, 3] [...] >And, of course, you can use the same dangling comma in making a tuple, >dictionary or set and who knows where else. >So, is that a feature you want warnings about? After all, a dangling >comma may simply mean you left something out and meant to add later? To my mind the killer argument for trailing commas is things like multiline lists, dicts or parameters. Example: def function( arg1=None, arg2=FOO, ): Imagine this for a comewhat extended set of parameters. Or similar for a big list or dict (eg a "table" as part of a class definition). By always including the trailing comma it is easier to insert or delete lines, and it reduces diff noise if you're using revision control (no commas flickering on and off in the diff output). Cheers, Cameron Simpson From rosuav at gmail.com Tue Dec 24 18:58:22 2019 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 25 Dec 2019 10:58:22 +1100 Subject: Lists And Extra Commas at end In-Reply-To: <00da01d5bab4$b011bdd0$10353970$@verizon.net> References: <00ba01d5baa3$df8db7a0$9ea926e0$@verizon.net> <20191224221228.GA23410@cskk.homeip.net> <00da01d5bab4$b011bdd0$10353970$@verizon.net> Message-ID: On Wed, Dec 25, 2019 at 10:50 AM Avi Gross via Python-list wrote: > > Cameron, > > I am not at all against the feature. I like it as my programming style is > like you describe. One entry per line indented at the same level, in > multiple languages. I often do graphics where I generate an image then > fine-tune additional parameters to get the effect I want. Some functions > take literally hundreds of options to adjust anything from text labels to > line types to colors and the high/low values on axes and so on. In languages > like R, I tend to put in a final entry with no trailing comma that is > something harmless that can be left in position and the same for the first > entry. Then my adding/deleting/editing of fields happens in the middle where > I always have a terminal comma. IOW you recreate the same feature by having a meaningless line at the end. Unideal but sometimes necessary. > My point is the convenience comes with a price for people who make a mistake > and are not told perhaps the dangling comma was a placeholder for something > to add that they forgot or ... People can learn. It's an extremely common idiom. If it's laid out vertically, include the trailing comma. It's not exactly hard to figure out. ChrisA From mail.python.org at marco.sulla.e4ward.com Tue Dec 24 19:20:25 2019 From: mail.python.org at marco.sulla.e4ward.com (Marco Sulla) Date: Wed, 25 Dec 2019 01:20:25 +0100 Subject: Lists And Extra Commas at end In-Reply-To: <00db01d5bab5$b9761c30$2c625490$@verizon.net> References: <00ba01d5baa3$df8db7a0$9ea926e0$.ref@verizon.net> <00ba01d5baa3$df8db7a0$9ea926e0$@verizon.net> <00db01d5bab5$b9761c30$2c625490$@verizon.net> Message-ID: On Wed, 25 Dec 2019 at 00:56, Avi Gross wrote: > I may not be understanding what you are objecting to I, sir, am objecting that I replied to a topic, and you answered to me, but in another topic. You could have respond to me in the correct topic, and then create this other one (that I'm not really interested). Anyway. About the extra comma, it's da**ed useful: a = ( 42, 1981, 8, 19, 23, ) If I have to comment out the last line, I can, **without having to remove the comma before**. And if I have to add another number at the end, I have not to remember to add the comma before. ######## The real problem is this one: a = 1, Unreadable and prone to subtle errors, because maybe you added the comma by mistake. Caution: Debugging Nightmares. The solution, IMHO? DeprecationWarning, use (1, ) Explicit is better blablabla. From mail.python.org at marco.sulla.e4ward.com Tue Dec 24 21:09:09 2019 From: mail.python.org at marco.sulla.e4ward.com (Marco Sulla) Date: Wed, 25 Dec 2019 03:09:09 +0100 Subject: Most elegant way to do something N times In-Reply-To: References: Message-ID: Anyway, from itertools recipes: def repeatfunc(func, times=None, *args): """Repeat calls to func with specified arguments. Example: repeatfunc(random.random) """ if times is None: return starmap(func, repeat(args)) return starmap(func, repeat(args, times)) On Tue, 24 Dec 2019 at 21:41, Marco Sulla wrote: > > On Tue, 24 Dec 2019 at 01:07, Chris Angelico wrote: > > On Tue, Dec 24, 2019 at 10:45 AM Marco Sulla <...> wrote: > > > ??? Excuse me, but why you needed to call the same function SIX times? This > > > seems to me not elegant in primis. > > > > > > Can you give us a practical example? > > > > File parsing. You read a section header and want to ignore that > > section, so you ignore the next 15 lines. > > mmap and find? > > > On Tue, 24 Dec 2019 at 01:35, DL Neil via Python-list > wrote: > > Taking the top/bottom six from a sorted list of occurrences. > > Slicing? From cs at cskk.id.au Tue Dec 24 21:40:14 2019 From: cs at cskk.id.au (Cameron Simpson) Date: Wed, 25 Dec 2019 13:40:14 +1100 Subject: Lists And Extra Commas at end In-Reply-To: References: Message-ID: <20191225024014.GA1946@cskk.homeip.net> On 25Dec2019 01:20, mail.python.org at marco.sulla.e4ward.com wrote: >About the extra comma, it's da**ed useful: [...] >The real problem is this one: > >a = 1, > >Unreadable and prone to subtle errors, because maybe you added the >comma by mistake. Caution: Debugging Nightmares. Hoo, yes. Only the other week I spent _way_ too long debugging exactly that mistake because I'd cut/pasted a parameter into some inline code. Suddenly a int was a tuple. With luck I'll recognise that mistake faster next time. Cheers, Cameron Simpson From jeff.gitlin at gmail.com Wed Dec 25 09:12:54 2019 From: jeff.gitlin at gmail.com (Jeff Gitlin) Date: Wed, 25 Dec 2019 06:12:54 -0800 (PST) Subject: Python 3.8 and := Message-ID: With the new operator := in Python 3.8 that allows you to do things like if ( x := f() ) == 1: Is there any reason to use just the assignment operator? From rosuav at gmail.com Wed Dec 25 09:22:31 2019 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 26 Dec 2019 01:22:31 +1100 Subject: Python 3.8 and := In-Reply-To: References: Message-ID: On Thu, Dec 26, 2019 at 1:16 AM Jeff Gitlin wrote: > > With the new operator := in Python 3.8 > that allows you to do things like > > if ( x := f() ) == 1: > > Is there any reason to use just the assignment operator? I don't understand the question. Are you asking why the normal "=" operator should be used? For starters, "=" is far more flexible than ":=" in terms of what it can assign to. ChrisA From manomugdha at gmail.com Wed Dec 25 07:43:44 2019 From: manomugdha at gmail.com (Manomugdha Biswas) Date: Wed, 25 Dec 2019 18:13:44 +0530 Subject: How to call C functions from Linux C executable and vice-versa Message-ID: Hi, I have a Linux C binary (executable, not .so file). i can update (add new files/folders if it is required) but after compilation i will get a binary executable file. I want to access/call few functions of this executable from python. for this i want to use cython interface (most probably cython is the best option!). Any other possible options are also acceptable. the example/documentation of cython i have seen till now, they talk about calling c functions from c library (.so file). can you please point me to a documentation/source code which talks about how to call c functions from c-executable? Regards, Mano From rosuav at gmail.com Wed Dec 25 11:44:00 2019 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 26 Dec 2019 03:44:00 +1100 Subject: How to call C functions from Linux C executable and vice-versa In-Reply-To: References: Message-ID: On Thu, Dec 26, 2019 at 3:41 AM Manomugdha Biswas wrote: > > Hi, > I have a Linux C binary (executable, not .so file). i can update (add new > files/folders if it is required) but after compilation i will get a binary > executable file. > I want to access/call few functions of this executable from python. for > this i want to use cython interface (most probably cython is the best > option!). Any other possible options are also acceptable. the > example/documentation of cython i have seen till now, they talk about > calling c functions from c library (.so file). > > can you please point me to a documentation/source code which talks about > how to call c functions from c-executable? > Once it's an executable, you can't call functions in it. So your options are either: 1) Go back to the source code and compile it differently. Use it as a library instead of an executable. 2) Run the program with parameters that make it do what you want, and then get a result back via stdout. The first one is almost certainly easier. You'll need to figure out how the project is structured, and probably learn a bit about the process of compiling C code to either an executable or a library. ChrisA From jeff.gitlin at gmail.com Wed Dec 25 12:15:21 2019 From: jeff.gitlin at gmail.com (Jeff Gitlin) Date: Wed, 25 Dec 2019 09:15:21 -0800 (PST) Subject: Python 3.8 and := In-Reply-To: References: Message-ID: <58fde3ea-ff62-47eb-bbe9-3ad9f6df9ada@googlegroups.com> On Wednesday, December 25, 2019 at 9:22:56 AM UTC-5, Chris Angelico wrote: > On Thu, Dec 26, 2019 at 1:16 AM Jeff Gitlin wrote: > > > > With the new operator := in Python 3.8 > > that allows you to do things like > > > > if ( x := f() ) == 1: > > > > Is there any reason to use just the assignment operator? > > I don't understand the question. Are you asking why the normal "=" > operator should be used? For starters, "=" is far more flexible than > ":=" in terms of what it can assign to. > > ChrisA Chris, Can you give me a few examples of what you can do with? = and not do with the new := ?? ? Thanks ? ? Jeff From rosuav at gmail.com Wed Dec 25 12:51:44 2019 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 26 Dec 2019 04:51:44 +1100 Subject: Python 3.8 and := In-Reply-To: <58fde3ea-ff62-47eb-bbe9-3ad9f6df9ada@googlegroups.com> References: <58fde3ea-ff62-47eb-bbe9-3ad9f6df9ada@googlegroups.com> Message-ID: On Thu, Dec 26, 2019 at 4:21 AM Jeff Gitlin wrote: > > On Wednesday, December 25, 2019 at 9:22:56 AM UTC-5, Chris Angelico wrote: > > On Thu, Dec 26, 2019 at 1:16 AM Jeff Gitlin wrote: > > > > > > With the new operator := in Python 3.8 > > > that allows you to do things like > > > > > > if ( x := f() ) == 1: > > > > > > Is there any reason to use just the assignment operator? > > > > I don't understand the question. Are you asking why the normal "=" > > operator should be used? For starters, "=" is far more flexible than > > ":=" in terms of what it can assign to. > > > > ChrisA > > Chris, > Can you give me a few examples of what you can do with = and not do with the new := ? https://www.python.org/dev/peps/pep-0572/#differences-between-assignment-expressions-and-assignment-statements ChrisA From jeff.gitlin at gmail.com Wed Dec 25 13:02:21 2019 From: jeff.gitlin at gmail.com (Jeff Gitlin) Date: Wed, 25 Dec 2019 10:02:21 -0800 (PST) Subject: Python 3.8 and := In-Reply-To: References: Message-ID: <759d915f-d3fd-4a34-a919-b72ecaa5e648@googlegroups.com> On Wednesday, December 25, 2019 at 9:13:09 AM UTC-5, Jeff Gitlin wrote: > With the new operator := in Python 3.8 > that allows you to do things like > > if ( x := f() ) == 1: > > Is there any reason to use just the assignment operator? ???????.. Thanks. That web page is exactly what I was looking for! From soyeomul at doraji.xyz Thu Dec 26 06:11:17 2019 From: soyeomul at doraji.xyz (=?utf-8?B?7Zmp67OR7Z2s?=) Date: Thu, 26 Dec 2019 20:11:17 +0900 Subject: [Q] __init__ in class Message-ID: in making class, why we should set "__init__"? so non-init class is useless? actually i did fail to understand __init__'s meaning. what is role of __init__ in class. sincerely, from python beginner. -- ^????? _????_ ?????_^))// From PythonList at DancesWithMice.info Thu Dec 26 13:23:12 2019 From: PythonList at DancesWithMice.info (DL Neil) Date: Fri, 27 Dec 2019 07:23:12 +1300 Subject: [Q] __init__ in class In-Reply-To: References: Message-ID: <48230401-ef4e-3457-cb71-945ca596def5@DancesWithMice.info> On 27/12/19 12:11 AM, ??? wrote: > in making class, why we should set "__init__"? so non-init class is > useless? actually i did fail to understand __init__'s meaning. what is > role of __init__ in class. > > sincerely, from python beginner. Please be aware that there is a Python Tutor Discussion List. If you are a beginner in Object-Oriented Programming, Wikipedia offers a multi-language, if dense introduction. This is complemented by their Class (computer programming) page. If you are new to Python's implementation then don't go past the Python Tutorial (in particular Section 9). There are plenty of web-tutorials discussion Python classes. HackerEarth's aligns nicely with the Wikipedia definitions (or the abstract terminology introduced in many ComSc texts and courses). WebRefs: Python Tutor: https://mail.python.org/mailman/listinfo/tutor Object-oriented programming: https://en.wikipedia.org/wiki/Object-oriented_programming Class (computer programming): https://en.wikipedia.org/wiki/Class_(computer_programming) The Python Tutorial: https://docs.python.org/3/tutorial/index.html HackerEarth: https://www.hackerearth.com/practice/python/object-oriented-programming/classes-and-objects-i/tutorial/ -- Regards =dn From Irv at furrypants.com Thu Dec 26 16:58:32 2019 From: Irv at furrypants.com (Irv Kalb) Date: Thu, 26 Dec 2019 13:58:32 -0800 Subject: [Q] __init__ in class In-Reply-To: References: Message-ID: <0A6BE303-2690-455C-8B71-964896E35A2A@furrypants.com> > On Dec 26, 2019, at 3:11 AM, ??? wrote: > > in making class, why we should set "__init__"? so non-init class is > useless? actually i did fail to understand __init__'s meaning. what is > role of __init__ in class. > > sincerely, from python beginner. > > -- > ^????? _????_ ?????_^))// > -- > https://mail.python.org/mailman/listinfo/python-list Whenever you create object from a class, you use the name of the class in a statement like this: myObject = MyClass() # "MyClass" would be the name of a class in your code When you run a line like the one above, Python looks to see if that class has a method named __init__(). If it does, then Python calls that method of the newly created object. In that method you typically put the initialization of any instance variables that you want to use in other methods, and include any additional code that you only want to run when an object is created from the class. While it is not required to have an __init__() method in a class, it is generally considered good practice to have one. As an example, let's say that you create a point class, where the Point has both an x and a yY value. Your class might look like this: class Point: def __init__(self, x, y): # When you create a Point object, you must pass in a value for x and y self.x = x self.y = y #Then you would probably have other methods like: def getX(self): return self.x def getY(self): return self.y def show(self): print('The values of this point are:', self.x, self.y) etc. Given a class like that, when you create an object from the Point class, you pass in values for x and y. The code in the __init__() method saves away those values into instance variables. Those instance variables can be used by other methods. In this example class, getX, getY, and show. Here is some sample code to create two Point objects, then ask them to show their contents: pointA = Point(10, 15) pointB = Point(-3, 38) pointA.show() printB.show() When run, this will print: The values of this point are: 10, 15 The values of this point are: -3, 38 Hope that gives you a start in the world of OOP. Irv From PythonList at DancesWithMice.info Thu Dec 26 19:56:04 2019 From: PythonList at DancesWithMice.info (DL Neil) Date: Fri, 27 Dec 2019 13:56:04 +1300 Subject: [Q] __init__ in class In-Reply-To: <0A6BE303-2690-455C-8B71-964896E35A2A@furrypants.com> References: <0A6BE303-2690-455C-8B71-964896E35A2A@furrypants.com> Message-ID: On 27/12/19 10:58 AM, Irv Kalb wrote: >> On Dec 26, 2019, at 3:11 AM, ??? wrote: >> in making class, why we should set "__init__"? so non-init class is >> useless? actually i did fail to understand __init__'s meaning. what is >> role of __init__ in class. ... > As an example, let's say that you create a point class, where the Point has both an x and a yY value. Your class might look like this: > > class Point: > def __init__(self, x, y): # When you create a Point object, you must pass in a value for x and y > self.x = x > self.y = y > > #Then you would probably have other methods like: > > def getX(self): > return self.x > > def getY(self): > return self.y > > def show(self): > print('The values of this point are:', self.x, self.y) > > etc. > > Given a class like that, when you create an object from the Point class, you pass in values for x and y. The code in the __init__() method saves away those values into instance variables. Those instance variables can be used by other methods. In this example class, getX, getY, and show. > > Here is some sample code to create two Point objects, then ask them to show their contents: > > pointA = Point(10, 15) > pointB = Point(-3, 38) > pointA.show() > printB.show() Is this a/the pythonic way to access attributes? It appears Java-like, or some other language transliterated into Python. Further, the naming convention suffers similar heritage. What was not shown is that in order to access the values of Point()'s x and y attribute, the following is needed: x_value = pointA.getX() which in and of itself is not too bad, if wordy and ugly; but what happens when we perform a vector addition? point_a_plus_b = Point( pointA.getX() + pointB.getX(), pointA.getY() + pointB.getY() ) Which is still not too shocking, but imagine a more complicated calculation, eg widen to include z- and w- dimensions or consider n-dimensional matrices... (there are optimised tools for such, but continuing with the post-er's example) Rather than a 'closed' view of objects ("encapsulation") Python maintains a more 'open' approach. "Getter" and "Setter" functions are put-aside in favor of direct-access: value = instance.attribute # 'getter' instance.attribute = value # 'setter' del instance.attribute # 'destructor' Now, imagine a class with ten attributes (for example) in Java (and similar). Before use, we would have to write ten getter-functions, ten setter-functions, and maybe (just to be 'complete') ten destructor-functions. Whereas in Python, nothing! Whatever you do, don't mention this to such coder-colleagues! We are able to code so much faster than they, and are considerably less restricted by our choice of language, thus promoting numerous efficacies! There are times when we can't allow 'just any value' to be applied to an attribute. For example, should a person's birth-date be recorded as ten-years from now? So, when we need to check or 'massage' data before use/recording, we might turn the attribute into a "property". This does give us the Python (and pythonic) version of getters-and-setters. Similarly, there are tactics which do enable data-hiding (or strict "encapsulation") for those occasions when it is needed...but now we're likely way beyond the OP's needs! WebRefs: (additional to those previously-posted in this thread) https://www.python-course.eu/python3_properties.php https://www.python.org/dev/peps/pep-0008/ -- Regards =dn From soyeomul at doraji.xyz Thu Dec 26 21:53:41 2019 From: soyeomul at doraji.xyz (=?utf-8?B?7Zmp67OR7Z2s?=) Date: Fri, 27 Dec 2019 11:53:41 +0900 Subject: [Q] __init__ in class References: Message-ID: Hellow Stefan^^^ > The class definition depends on the requirement > specification for the class. > > ??? ??? ???? ?? ?? ??? ?? ????. Now i got it, understood, thanks a lot^^^ Sincerely, -- ^????? _????_ ?????_^))// From soyeomul at doraji.xyz Thu Dec 26 21:56:19 2019 From: soyeomul at doraji.xyz (=?utf-8?B?7Zmp67OR7Z2s?=) Date: Fri, 27 Dec 2019 11:56:19 +0900 Subject: [Q] __init__ in class References: <48230401-ef4e-3457-cb71-945ca596def5@DancesWithMice.info> Message-ID: Hellow DL^^^ > Please be aware that there is a Python Tutor Discussion List. Thanks for advice^^^ Sincerely, -- ^????? _????_ ?????_^))// From soyeomul at doraji.xyz Thu Dec 26 21:59:33 2019 From: soyeomul at doraji.xyz (=?utf-8?B?7Zmp67OR7Z2s?=) Date: Fri, 27 Dec 2019 11:59:33 +0900 Subject: [Q] __init__ in class References: <0A6BE303-2690-455C-8B71-964896E35A2A@furrypants.com> Message-ID: Irv Kalb writes: > myObject = MyClass() # "MyClass" would be the name of a class in your code > > When you run a line like the one above, Python looks to see if that > class has a method named __init__(). If it does, then Python calls > that method of the newly created object. Very clean comments, thank you Irv^^^ INDEED. Sincerely, -- ^????? _????_ ?????_^))// From vid_tech at yahoo.com Fri Dec 27 01:29:39 2019 From: vid_tech at yahoo.com (Kishor Soni) Date: Fri, 27 Dec 2019 06:29:39 +0000 (UTC) Subject: Error in python installation References: <66413736.2558666.1577428179364.ref@mail.yahoo.com> Message-ID: <66413736.2558666.1577428179364@mail.yahoo.com> Hello I have Intel DH61 series mother board, with Windows7 64bit + SP1 + some debugging components ( as shown in microsoft website, as service pack for W7) however when I try to install current version of python, it displays error. See the screenshot attached herewith. (board configuration displays in background window) Kishor Soni, Vid Technology, Rajkot - Gujarat. URL: www.vidtech.co.in, Landline: 0281-2225527 Cell: 093757 25527 From auriocus at gmx.de Fri Dec 27 15:59:48 2019 From: auriocus at gmx.de (Christian Gollwitzer) Date: Fri, 27 Dec 2019 21:59:48 +0100 Subject: scrollable Frame object In-Reply-To: References: <66620145-f0ec-4752-8d69-eae3474481f8@googlegroups.com> Message-ID: Hi David, Am 24.12.19 um 13:48 schrieb david.isal2 at gmail.com: > thank you for your answer. I downloaded and had a look at your library, it is a great solution to use a canvas-free scrollable frame. but it is in pure tcl, how do I use this with python tkinter ? Yes, because you asked this in the Tcl/Tk newsgroup. We use Tcl directly here, not via Python Tkinter. > Although my initial request is that the frame object is not scrollable by default. > so My request for the tcl/tk developers would be, why not make the object frame scrollbale by default, in order to attach a scrollbar to it? > I believe it is a typical need. This may well be. For that, the developers of Tkinter (distinct from the developers of Tk and Tcl) have to include a scrollable frame into Tkinter. There is unfortunately a number of extension packages which exist for Tk, but which are not wrapped by a standard installatino of Tkinter for Python, despite them being extremely useful / not even possible otherwise. Among those are: * Tablelist - a great tree and multi-column widget. Much more versatile then the built-in treeview * TkDnD - native drag'n'drop support, works on all 3 OSes * tkPath - a canvas replacement with "modern" 2D features (anti-aliasing, image scaling/rotation, alpha blending) * pdf4tcl, generate PDF images from a canvas drawing * tkTable, spreadsheet-like widget * tktreectrl, yet another extremely powerful treeview / explorer like widget Building support for these packages would not be that hard, but it must be done by the Python community. Christian (X-post & Fup to clp) From jfong at ms4.hinet.net Fri Dec 27 22:20:44 2019 From: jfong at ms4.hinet.net (jfong at ms4.hinet.net) Date: Fri, 27 Dec 2019 19:20:44 -0800 (PST) Subject: Does the argparse generate a wrong help message? Message-ID: The codes in test.py are: --------- import argparse parser = argparse.ArgumentParser() parser.add_argument('--foo', nargs='?', help='foo help') parser.add_argument('--goo', nargs=1, help='goo help') args = parser.parse_args() print(args.foo, args.goo) --------- But I get the following result: --------- D:\Works\Python>py test.py -h usage: test.py [-h] [--foo [FOO]] [--goo GOO] optional arguments: -h, --help show this help message and exit --foo [FOO] foo help --goo GOO goo help D:\Works\Python>py test.py --foo 1 --goo 2 1 ['2'] --------- It seems to me that the help message should be: usage: test.py [-h] [--foo FOO] [--goo [GOO]] Do I had missed something? --Jach From rosuav at gmail.com Fri Dec 27 22:30:22 2019 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 28 Dec 2019 14:30:22 +1100 Subject: Does the argparse generate a wrong help message? In-Reply-To: References: Message-ID: On Sat, Dec 28, 2019 at 2:26 PM wrote: > > The codes in test.py are: > --------- > import argparse > parser = argparse.ArgumentParser() > parser.add_argument('--foo', nargs='?', help='foo help') > parser.add_argument('--goo', nargs=1, help='goo help') > args = parser.parse_args() > print(args.foo, args.goo) > --------- > > But I get the following result: > --------- > D:\Works\Python>py test.py -h > usage: test.py [-h] [--foo [FOO]] [--goo GOO] > > optional arguments: > -h, --help show this help message and exit > --foo [FOO] foo help > --goo GOO goo help > > D:\Works\Python>py test.py --foo 1 --goo 2 > 1 ['2'] > --------- > > It seems to me that the help message should be: > usage: test.py [-h] [--foo FOO] [--goo [GOO]] > > Do I had missed something? > Can you elaborate on why you expect this? You've declared one of them to have a single mandatory argument, and the other a single optional argument. This corresponds to what I'm seeing. ChrisA From jfong at ms4.hinet.net Fri Dec 27 23:08:16 2019 From: jfong at ms4.hinet.net (jfong at ms4.hinet.net) Date: Fri, 27 Dec 2019 20:08:16 -0800 (PST) Subject: Does the argparse generate a wrong help message? In-Reply-To: References: Message-ID: <63c01a3b-6e47-48e7-9405-d144371b54d6@googlegroups.com> Chris Angelico? 2019?12?28???? UTC+8??11?30?47???? > On Sat, Dec 28, 2019 at 2:26 PM wrote: > > > > The codes in test.py are: > > --------- > > import argparse > > parser = argparse.ArgumentParser() > > parser.add_argument('--foo', nargs='?', help='foo help') > > parser.add_argument('--goo', nargs=1, help='goo help') > > args = parser.parse_args() > > print(args.foo, args.goo) > > --------- > > > > But I get the following result: > > --------- > > D:\Works\Python>py test.py -h > > usage: test.py [-h] [--foo [FOO]] [--goo GOO] > > > > optional arguments: > > -h, --help show this help message and exit > > --foo [FOO] foo help > > --goo GOO goo help > > > > D:\Works\Python>py test.py --foo 1 --goo 2 > > 1 ['2'] > > --------- > > > > It seems to me that the help message should be: > > usage: test.py [-h] [--foo FOO] [--goo [GOO]] > > > > Do I had missed something? > > > > Can you elaborate on why you expect this? You've declared one of them > to have a single mandatory argument, and the other a single optional > argument. This corresponds to what I'm seeing. > > ChrisA So the square bracket means optional, not list? My misunderstanding:-( --Jach From Richard at Damon-Family.org Fri Dec 27 23:18:28 2019 From: Richard at Damon-Family.org (Richard Damon) Date: Fri, 27 Dec 2019 23:18:28 -0500 Subject: Does the argparse generate a wrong help message? In-Reply-To: <63c01a3b-6e47-48e7-9405-d144371b54d6@googlegroups.com> References: <63c01a3b-6e47-48e7-9405-d144371b54d6@googlegroups.com> Message-ID: On 12/27/19 11:08 PM, jfong at ms4.hinet.net wrote: > Chris Angelico? 2019?12?28???? UTC+8??11?30?47???? >> On Sat, Dec 28, 2019 at 2:26 PM wrote: >>> The codes in test.py are: >>> --------- >>> import argparse >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo', nargs='?', help='foo help') >>> parser.add_argument('--goo', nargs=1, help='goo help') >>> args = parser.parse_args() >>> print(args.foo, args.goo) >>> --------- >>> >>> But I get the following result: >>> --------- >>> D:\Works\Python>py test.py -h >>> usage: test.py [-h] [--foo [FOO]] [--goo GOO] >>> >>> optional arguments: >>> -h, --help show this help message and exit >>> --foo [FOO] foo help >>> --goo GOO goo help >>> >>> D:\Works\Python>py test.py --foo 1 --goo 2 >>> 1 ['2'] >>> --------- >>> >>> It seems to me that the help message should be: >>> usage: test.py [-h] [--foo FOO] [--goo [GOO]] >>> >>> Do I had missed something? >>> >> Can you elaborate on why you expect this? You've declared one of them >> to have a single mandatory argument, and the other a single optional >> argument. This corresponds to what I'm seeing. >> >> ChrisA > So the square bracket means optional, not list? My misunderstanding:-( > > --Jach Yes, the normal notation is [x] indicates that x is optionial -- Richard Damon From nt_mahmood at yahoo.com Sat Dec 28 00:29:53 2019 From: nt_mahmood at yahoo.com (Mahmood Naderan) Date: Sat, 28 Dec 2019 08:59:53 +0330 Subject: Grepping words for match in a file Message-ID: <47lC1y0nHDznXmY@mail.python.org> Hi I have some lines in a text file like ADD R1, R2 ADD3 R4, R5, R6 ADD.MOV R1, R2, [0x10] If I grep words with this code for line in fp: if my_word in line: Then if my_word is "ADD", I get 3 matches. However, if I grep word with this code for line in fp: for word in line.split(): if my_word == word: Then I get only one match which is ADD R1. R2. Actually I want to get 2 matches. ADD R1, R2 and ADD.MOV R1, R2, [0x10] because these two lines are actually "ADD" instructions. However, "ADD3" is something else. How can I fix the code for that purpose? -- Regards, Mahmood From smitla at yandex.com Sat Dec 28 05:52:54 2019 From: smitla at yandex.com (L A Smit) Date: Sat, 28 Dec 2019 12:52:54 +0200 Subject: Front end Message-ID: Hi Don't know if this is the correct subject but i want a program like an invoice, You make an invoice and save it and the next one is ready to use. I am completely new to programming and want this program for myself. I want to use python to do it. Have already build the program but don't know how to put it in usable format. I hope you understand what i mean. Ex: Input. Quote Nr: Client: Product: Then there will be costs and in the end a cost per product. In the end a save button to save quote with input from line 1 and 2. Then the template is ready for next input. I understand that there is probably hundreds of these programs but to teach myself i want to wright my own program and then i can update it when needed. Thx L Smit From jsf80238 at gmail.com Sat Dec 28 11:05:33 2019 From: jsf80238 at gmail.com (Jason Friedman) Date: Sat, 28 Dec 2019 09:05:33 -0700 Subject: Grepping words for match in a file In-Reply-To: <47lC1y0nHDznXmY@mail.python.org> References: <47lC1y0nHDznXmY@mail.python.org> Message-ID: > > > I have some lines in a text file like > ADD R1, R2 > ADD3 R4, R5, R6 > ADD.MOV R1, R2, [0x10] > .... Actually I want to get 2 matches. ADD R1, R2 and ADD.MOV R1, R2, [0x10] > because these two lines are actually "ADD" instructions. However, "ADD3" is > something else. > >>> s = """ADD R1, R2 ... ADD3 R4, R5, R6 ... ADD.MOV R1, R2, [0x10]""" >>> import re >>> pattern = re.compile(r"^ADD\b.+") >>> for line in s.splitlines(): ... if re.search(pattern, line): ... print(line) ... ADD R1, R2 ADD.MOV R1, R2, [0x10] From 2QdxY4RzWzUUiLuE at potatochowder.com Sat Dec 28 11:05:57 2019 From: 2QdxY4RzWzUUiLuE at potatochowder.com (Dan Sommers) Date: Sat, 28 Dec 2019 11:05:57 -0500 Subject: Grepping words for match in a file In-Reply-To: <47lC1y0nHDznXmY@mail.python.org> References: <47lC1y0nHDznXmY@mail.python.org> Message-ID: <64a7f92e-3899-aa9e-ec5a-068c0f196994@potatochowder.com> On 12/28/19 12:29 AM, Mahmood Naderan via Python-list wrote: > Hi > I have some lines in a text file like > ADD R1, R2 > ADD3 R4, R5, R6 > ADD.MOV R1, R2, [0x10] > If I grep words with this code > for line in fp: > if my_word in line: > Then if my_word is "ADD", I get 3 matches. However, if I grep word with this code > for line in fp: > for word in line.split(): > if my_word == word: > Then I get only one match which is ADD R1. R2. > Actually I want to get 2 matches. ADD R1, R2 and ADD.MOV R1, R2, [0x10] because these two lines are actually "ADD" instructions. However, "ADD3" is something else. > How can I fix the code for that purpose? Grep's "words" are different from what line.split() returns: >>> 'ADD.MOV R1, R2, [0x10]'.split() ['ADD.MOV', 'R1,', 'R2,', '[0x10]'] If you want to match "ADD" instructions, you'll have to use word.startswith or some other method to separate the "ADD" from the ".MOV." Dan From 2QdxY4RzWzUUiLuE at potatochowder.com Sat Dec 28 11:14:14 2019 From: 2QdxY4RzWzUUiLuE at potatochowder.com (Dan Sommers) Date: Sat, 28 Dec 2019 11:14:14 -0500 Subject: Grepping words for match in a file In-Reply-To: <47lC1y0nHDznXmY@mail.python.org> References: <47lC1y0nHDznXmY@mail.python.org> Message-ID: On 12/28/19 12:29 AM, Mahmood Naderan via Python-list wrote: > Hi > I have some lines in a text file like > ADD R1, R2 > ADD3 R4, R5, R6 > ADD.MOV R1, R2, [0x10] > If I grep words with this code > for line in fp: > if my_word in line: > Then if my_word is "ADD", I get 3 matches. However, if I grep word with this code > for line in fp: > for word in line.split(): > if my_word == word: > Then I get only one match which is ADD R1. R2. > Actually I want to get 2 matches. ADD R1, R2 and ADD.MOV R1, R2, [0x10] because these two lines are actually "ADD" instructions. However, "ADD3" is something else. > How can I fix the code for that purpose? (1) word.startswith() won't solve your problem. Expliticly checking the character after the second "D" would. You'll have to determine which characters are or aren't part of the instruction. A complete solution probably depends on what else you are or will look for in the future. (2) That looks like a programming language (88000? POWER?). Watch out for comments containing the word ADD, too. From tjreedy at udel.edu Sat Dec 28 15:48:23 2019 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 28 Dec 2019 15:48:23 -0500 Subject: Front end In-Reply-To: References: Message-ID: On 12/28/2019 5:52 AM, L A Smit wrote: > Hi > > Don't know if this is the correct subject but i want a program like an > invoice, You make an invoice and save it and the next one is ready to > use. I am completely new to programming and want this program for myself. > > > I want to use python to do it. Have already build the program but don't > know how to put it in usable format. I hope you understand what i mean. > > Ex: Input. > > Quote Nr: > > Client: > > Product: > > Then there will be costs and in the end a cost per product. > > In the end a save button to save quote with input from line 1 and 2. > > Then the template is ready for next input. Start with the simplest thing that works, which would be something like: quotenum = input('Quote Nr: ') client = input('Client: ') ... invoice = f"""\ Quote Nr: {quotenum} Client: {client} ... """ with open('quotenum'+'client', mode='w') as f: f.write(invoice) Then do something fancier if you want. 1. GUI form with tkinter. 2. Generate input statements or GUI form from the invoice template string. -- Terry Jan Reedy From PythonList at DancesWithMice.info Sat Dec 28 17:35:57 2019 From: PythonList at DancesWithMice.info (DL Neil) Date: Sun, 29 Dec 2019 11:35:57 +1300 Subject: Friday Finking: Source code organisation Message-ID: Is it helpful to, and thus, do you have a style/convention for ordering the methods within each class in your code? Python's "honking great idea" of namespaces enables us to gather functions/methods under a single name-umbrella, thereby avoiding name-clashes/name-space 'pollution'. [Oh yeah!] Thus, if we collect a bunch of module-functions, we refer to them collectively by module-name/import-name, eg import collected_functions_module as cfm ... cfm.make_it_happen() cfm.make_it_better() Similarly, methods within a class may be accessed either as class-methods, or through instances (as required). [am assuming no code example necessary] A major difference however, is that if our mythical collection of module-functions has an internal-reference, eg b() requires a(), then function a() MUST exist, ie be defined, 'before' function b(). Whereas a class's methods may be defined in any (complete) sequence. In the ?good? old days, eg COBOL - or more logically: when source code was most-usually viewed as printouts and the prevailing coding-style was for largely monolithic program(me)s; we would often number subroutines (initially in tens) to better navigate long listings of source-code, and more easily locate the subroutine's code from some reference-point. In data-terms, we might even create an abbreviation of a record's name and use that as an hyphenated-prefix for all of its data-fields' names. Today, such artifice is unnecessary, given that code-editors (normally) offer one-click access from reference to source - even spanning multiple Python modules. [another, "oh yeah"!] So, do you have an orderly method [hah!] for presenting/locating class-methods (and module-functions) within your code? - why bother, the editor does 'the heavy lifting' - dunders to the fore - alphanumeric sequence by name - order of appearance/use in 'mainline code' - as they sprang to mind during TDD-creation - most-used first, least-used last - my code 'at the top', their stuff later... - names of Monty Python characters by TV appearance date or, - some combination of ideas and, - how do you vary the above when dependencies intrude? -- Regards, =dn From rosuav at gmail.com Sat Dec 28 17:49:06 2019 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 29 Dec 2019 09:49:06 +1100 Subject: Friday Finking: Source code organisation In-Reply-To: References: Message-ID: On Sun, Dec 29, 2019 at 9:37 AM DL Neil via Python-list wrote: > > Is it helpful to, and thus, do you have a style/convention for ordering > the methods within each class in your code? > > A major difference however, is that if our mythical collection of > module-functions has an internal-reference, eg b() requires a(), then > function a() MUST exist, ie be defined, 'before' function b(). Whereas a > class's methods may be defined in any (complete) sequence. > > So, do you have an orderly method [hah!] for presenting/locating > class-methods (and module-functions) within your code? > > - why bother, the editor does 'the heavy lifting' > - dunders to the fore > - alphanumeric sequence by name > - order of appearance/use in 'mainline code' > - as they sprang to mind during TDD-creation > - most-used first, least-used last > - my code 'at the top', their stuff later... > - names of Monty Python characters by TV appearance date > or, > - some combination of ideas > and, > - how do you vary the above when dependencies intrude? > "Define before use" is a broad principle that I try to follow, even when the code itself doesn't mandate this. This generally means that "if name is main" is the very last thing in the file, and if there's a main() function or equivalent, that's usually just before that. Any metaprogramming goes right at the top; sometimes this is mandated (if I write a decorator function, it has to be above the functions it's decorating), but even if it's not, metaprogramming goes before the mainline. Other than that, I don't have any consistent logic other than a loose idea of trying to keep related things together. ChrisA From Richard at Damon-Family.org Sat Dec 28 18:39:42 2019 From: Richard at Damon-Family.org (Richard Damon) Date: Sat, 28 Dec 2019 18:39:42 -0500 Subject: Friday Finking: Source code organisation In-Reply-To: References: Message-ID: <3d4e9955-f83e-4641-9f77-1fcab809df4f@Damon-Family.org> On 12/28/19 5:35 PM, DL Neil via Python-list wrote: > Is it helpful to, and thus, do you have a style/convention for > ordering the methods within each class in your code? > > > Python's "honking great idea" of namespaces enables us to gather > functions/methods under a single name-umbrella, thereby avoiding > name-clashes/name-space 'pollution'. [Oh yeah!] > > Thus, if we collect a bunch of module-functions, we refer to them > collectively by module-name/import-name, eg > > ????import collected_functions_module as cfm > ????... > ????cfm.make_it_happen() > ????cfm.make_it_better() > > Similarly, methods within a class may be accessed either as > class-methods, or through instances (as required). > [am assuming no code example necessary] > > A major difference however, is that if our mythical collection of > module-functions has an internal-reference, eg b() requires a(), then > function a() MUST exist, ie be defined, 'before' function b(). Whereas > a class's methods may be defined in any (complete) sequence. If module function b calls function a in the same module, then a has to exist when b is CALLED, not before it is defined. Thus it is very possible to get a multi-function recursion where b() calls a() which then calls b(). (The existence of items mentioned within a function are looked up until that statement as actually executed) as an example, in a module: def b(): ? a() def a(): ? pass is perfectly valid. Modules are a bit different, if module b needs resources from module a, it needs to import module a before it can use them. If module a also needs resources from module b, and imports it, then stuff from b might not be available while doing the running of module a that is defining the items in a. -- Richard Damon From greg.ewing at canterbury.ac.nz Sat Dec 28 18:42:37 2019 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 29 Dec 2019 12:42:37 +1300 Subject: Friday Finking: Source code organisation In-Reply-To: References: Message-ID: On 29/12/19 11:35 am, DL Neil wrote: > if our mythical collection of > module-functions has an internal-reference, eg b() requires a(), then > function a() MUST exist, ie be defined, 'before' function b(). Not true in Python -- a() must exist by the time b() is run, but they can be written in the file in either order, -- Greg From greg.ewing at canterbury.ac.nz Sat Dec 28 18:52:54 2019 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 29 Dec 2019 12:52:54 +1300 Subject: Friday Finking: Source code organisation In-Reply-To: References: Message-ID: On 29/12/19 11:49 am, Chris Angelico wrote: > "Define before use" is a broad principle that I try to follow, even > when the code itself doesn't mandate this. I tend to do this too, although it's probably just a habit carried over from languages such as Pascal and C where you have to go out of your way to get things in a different order. But strangely, I tend to do the opposite for methods of a class. I don't really know why. My instinctive idea of the "right" ordering just seems to flip over somehow between modules and classes. -- Greg From rosuav at gmail.com Sat Dec 28 20:12:19 2019 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 29 Dec 2019 12:12:19 +1100 Subject: Friday Finking: Source code organisation In-Reply-To: References: Message-ID: On Sun, Dec 29, 2019 at 10:56 AM Greg Ewing wrote: > > On 29/12/19 11:49 am, Chris Angelico wrote: > > "Define before use" is a broad principle that I try to follow, even > > when the code itself doesn't mandate this. > > I tend to do this too, although it's probably just a habit > carried over from languages such as Pascal and C where you > have to go out of your way to get things in a different > order. IMO it makes the code easier to navigate even when it's not strictly necessary. As others have said, Python mandates that the functions be defined before they're CALLED, but I find that it's worth being stricter. As a general rule, any global name in my code (that includes constants, module-level functions, etc, etc) will have its definition as the textually-first instance of that name. Or, at worst, the first instance will be a comment immediately above the definition. > But strangely, I tend to do the opposite for methods of > a class. I don't really know why. My instinctive idea of > the "right" ordering just seems to flip over somehow > between modules and classes. That's curious. I have a similar sort of inversion in that __init__ is usually going to be up the top (even though main() would be at the bottom), although not a complete inversion: anything that __init__ calls will still go above it. ChrisA From 2QdxY4RzWzUUiLuE at potatochowder.com Sat Dec 28 20:13:27 2019 From: 2QdxY4RzWzUUiLuE at potatochowder.com (Dan Sommers) Date: Sat, 28 Dec 2019 20:13:27 -0500 Subject: Friday Finking: Source code organisation In-Reply-To: References: Message-ID: <62196530-433e-5414-7262-1398c72465e8@potatochowder.com> On 12/28/19 6:52 PM, Greg Ewing wrote: > On 29/12/19 11:49 am, Chris Angelico wrote: >> "Define before use" is a broad principle that I try to follow, even >> when the code itself doesn't mandate this. > But strangely, I tend to do the opposite for methods of a class. I > don't really know why. My instinctive idea of the "right" ordering > just seems to flip over somehow between modules and classes. Me, too. :-/ Could be one of two reasons: (1) putting the public API at the top and pushing the internal/helper functions towards the bottom (whether program, library, or class); or (2) writing too much FORTRAN and/or BASIC early on, where execution begins at the top, whether you want it to or not Who puts a class's __init__ function anywhere except the very top of a class definition (doc strings and class attributes that aren't "def"ed notwithstanding), even when it calls some other method in the class? Dan From python.list at tim.thechases.com Sat Dec 28 20:19:04 2019 From: python.list at tim.thechases.com (Tim Chase) Date: Sat, 28 Dec 2019 19:19:04 -0600 Subject: Friday Finking: Source code organisation In-Reply-To: References: Message-ID: <20191228191904.5327f8c5@bigbox.attlocal.net> On 2019-12-29 12:52, Greg Ewing wrote: > On 29/12/19 11:49 am, Chris Angelico wrote: > > "Define before use" is a broad principle that I try to follow, > > even when the code itself doesn't mandate this. > > I tend to do this too, although it's probably just a habit > carried over from languages such as Pascal and C where you > have to go out of your way to get things in a different > order. Apparently I'm not alone in my Pascal/C-derived habits of define-before-use. Inside a class, I tend to roughly follow __new__ (if present) __init__ other dunder methods subsequent methods alphabetically -tkc From PythonList at DancesWithMice.info Sat Dec 28 21:32:18 2019 From: PythonList at DancesWithMice.info (DL Neil) Date: Sun, 29 Dec 2019 15:32:18 +1300 Subject: Friday Finking: Source code organisation In-Reply-To: References: Message-ID: <88ce9ccc-cb4f-b081-d536-649745b6b2a3@DancesWithMice.info> >>> "Define before use" is a broad principle that I try to follow, even >>> when the code itself doesn't mandate this. > > IMO it makes the code easier to navigate even when it's not strictly > necessary. As others have said, Python mandates that the functions be > defined before they're CALLED, but I find that it's worth being > stricter. As a general rule, any global name in my code (that includes > constants, module-level functions, etc, etc) will have its definition > as the textually-first instance of that name. Or, at worst, the first > instance will be a comment immediately above the definition. ('Anonymised' because this is not 'aimed' at any particular individual) I was somewhat amused at being 'corrected' in the "defined" cf "called" discussion. The 'corrections' are more precise. However such distinctions are machine-related, ie how does the Python interpreter behave? The central topic of this (ideally, thought-provoking) discussion, is on the person (not the machine). Yes, the author. Although, if you subscribe to the philosophy that program(me)s should be written with human-readers in-mind, then the likes of "beauty is in the eye of the beholder" and "readability counts", apply! Various 'gurus', probably even before the Agile/Gang-of-Four/OOP, have talked of code craftsmanship having a number of characteristics. One of which is that *my* code should appear pretty-much as *you* expect the solution. When we were asked to discuss this, over-the-teacups (on Friday, hence the "FF" moniker) the young fellow's question was phrased "what do you guys expect [from me]". (yes, he was quoting/abstracting from 'Uncle Bob' and "Clean Code" [Prentice-Hall, 2009]). We have a number of 'what do you expect'-s in Python. For example, there is no "law" saying that we must use "self" within classes. Other languages use similar but alternate terminology, eg "this". I could decide to use "dn_was_here" and the interpreter will be perfectly happy - but because you don't expect it and would be made to (unnecessarily) comprehend, I doubt it would promote our friendship/mutual-respect! So, do you hold any 'expectation' relating to the organisation of class methods? -- Regards =dn From PythonList at DancesWithMice.info Sat Dec 28 21:43:46 2019 From: PythonList at DancesWithMice.info (DL Neil) Date: Sun, 29 Dec 2019 15:43:46 +1300 Subject: Friday Finking: Source code organisation In-Reply-To: <20191228191904.5327f8c5@bigbox.attlocal.net> References: <20191228191904.5327f8c5@bigbox.attlocal.net> Message-ID: <9d3049aa-0308-30d6-6aa1-648b50ab640c@DancesWithMice.info> On 29/12/19 2:19 PM, Tim Chase wrote: > On 2019-12-29 12:52, Greg Ewing wrote: >> I tend to do this too, although it's probably just a habit >> carried over from languages such as Pascal and C where you >> have to go out of your way to get things in a different >> order. > > Apparently I'm not alone in my Pascal/C-derived habits of > define-before-use. I didn't expect to be the only one who would bring 'prior experience' into it... > Inside a class, I tend to roughly follow > __new__ (if present) > __init__ > other dunder methods > subsequent methods alphabetically +1 Should __new__() and __init__() break with convention because its/their method/function signature is (really) part of the class's signature, and thus there's a need for proximity? (aka is "expected") -- Regards =dn From PythonList at DancesWithMice.info Sat Dec 28 21:54:22 2019 From: PythonList at DancesWithMice.info (DL Neil) Date: Sun, 29 Dec 2019 15:54:22 +1300 Subject: Grepping words for match in a file In-Reply-To: References: <47lC1y0nHDznXmY@mail.python.org> Message-ID: <68f01e33-ec46-4eb1-7ad5-46847cea5760@DancesWithMice.info> On 29/12/19 5:14 AM, Dan Sommers wrote: > On 12/28/19 12:29 AM, Mahmood Naderan via Python-list wrote: >> Hi >> I have some lines in a text file like >> ADD R1, R2 >> ADD3 R4, R5, R6 >> ADD.MOV R1, R2, [0x10] >> If I grep words with this code >> for line in fp: >> ???? if my_word in line: >> Then if my_word is "ADD", I get 3 matches. However, if I grep word >> with this code >> for line in fp: >> ???? for word in line.split(): >> ???????? if my_word == word: >> Then I get only one match which is ADD R1. R2. >> Actually I want to get 2 matches. ADD R1, R2 and ADD.MOV R1, R2, >> [0x10] because these two lines are actually "ADD" instructions. >> However, "ADD3" is something else. >> How can I fix the code for that purpose? > (1) word.startswith() won't solve your problem.? Expliticly > checking the character after the second "D" would.? You'll > have to determine which characters are or aren't part of > the instruction.? A complete solution probably depends on > what else you are or will look for in the future. > > (2) That looks like a programming language (88000? POWER?). > Watch out for comments containing the word ADD, too. For which reason, and given its finite and limited nature, I'd prefer to be explicit:- Assuming "ADD" is only a sample of the total task, perhaps it will not be too impractical to create categories of instruction: ADD_CATEGORY = [ "ADD", "ADD.MOV" ] etc Now, you can perform an "in" on the first token: if line.split()[ 0 ] in ADD_CATEGORY: # this command one of those in the ADD category... In this way, rather than relying upon (actually *not* being able to rely upon) similarities of appearance, you have complete control to limit which commands fit into which category/ies... -- Regards =dn From cs at cskk.id.au Sat Dec 28 23:49:27 2019 From: cs at cskk.id.au (Cameron Simpson) Date: Sun, 29 Dec 2019 15:49:27 +1100 Subject: Friday Finking: Source code organisation In-Reply-To: References: Message-ID: <20191229044927.GA50701@cskk.homeip.net> On 29Dec2019 09:49, Chris Angelico wrote: >"Define before use" is a broad principle that I try to follow, even >when the code itself doesn't mandate this. This generally means that >"if name is main" is the very last thing in the file, and if there's a >main() function or equivalent, that's usually just before that. Any >metaprogramming goes right at the top; sometimes this is mandated (if >I write a decorator function, it has to be above the functions it's >decorating), but even if it's not, metaprogramming goes before the >mainline. For main, i have the opposite habit. If a module has a main() function for command line use I usually want that right up the front: #!/usr/bin/env python3 .... import... def main(argv=None): ... main command line ... classes, functions, etc if __name__ == '__main__': sys.exit(main(sys.argv)) My reasoning here is that I want the main programme obvious up front. But then I loosely follow "define before use" after that. Cheers, Cameron Simpson From cs at cskk.id.au Sat Dec 28 23:55:31 2019 From: cs at cskk.id.au (Cameron Simpson) Date: Sun, 29 Dec 2019 15:55:31 +1100 Subject: Friday Finking: Source code organisation In-Reply-To: <20191228191904.5327f8c5@bigbox.attlocal.net> References: <20191228191904.5327f8c5@bigbox.attlocal.net> Message-ID: <20191229045531.GA61883@cskk.homeip.net> On 28Dec2019 19:19, Tim Chase wrote: >Inside a class, I tend to roughly follow > > __new__ (if present) > __init__ > other dunder methods > subsequent methods alphabetically I put the factory methods up near __init__, ahead of other methods. But after most dunders. Here I mean things like this: class Foo: def __init__(....): @classmethod def from_bytes(cls, bs, ...): ... parse out of `bs` and then finish with something like ... return cls(parsed-stuff) which calls the "conventional" __init__ as normal. And I often group methods of similar purpose together, rather than alphabetically. For example, if there's a few methods for transribing the object, they might land together. Cheers, Cameron Simpson From drsalists at gmail.com Sun Dec 29 13:19:19 2019 From: drsalists at gmail.com (Dan Stromberg) Date: Sun, 29 Dec 2019 10:19:19 -0800 Subject: With distutils, "optional" C extension module error is fatal Message-ID: Hi folks. I'm putting a little time into getting my treap module (like a dict, but always sorted by key) working with distutils. I want it to be able to compile and install the Cython version from an included .c file, or to fall back on a pure python version if that fails. I'm currently using: setup( name='treap', py_modules=[ 'treap', 'py_treap', 'nest', ], # ext_modules=cythonize("pyx_treap.pyx"), ext_modules=[Extension('pyx_treap', ['pyx_treap.c'], optional=True)], # setup_requires=[ # 'Cython' # ], version=version, description='Python implementation of treaps, ) ...plus a few more options to setup that don't seem relevant. If I run python3.6 setup.py build, I get: running build running build_py creating build creating build/lib.linux-x86_64-3.6 copying treap.py -> build/lib.linux-x86_64-3.6 copying py_treap.py -> build/lib.linux-x86_64-3.6 copying nest.py -> build/lib.linux-x86_64-3.6 running build_ext building 'pyx_treap' extension creating build/temp.linux-x86_64-3.6 x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -c pyx_treap.c -o build/temp.linux-x86_64-3.6/pyx_treap.o pyx_treap.c:4:10: fatal error: Python.h: No such file or directory #include "Python.h" ^~~~~~~~~~ compilation terminated. error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 So it seems like despite using Extension('pyx_treap', ['pyx_treap.c'], optional=True), the cython version is not optional. Am I missing something? Why doesn't it just continue with the pure python modules? I realize I could install python3.6-dev or similar to eliminate the error, but I don't want to assume the user knows that. Thanks! From dfnsonfsduifb at gmx.de Sun Dec 29 17:02:48 2019 From: dfnsonfsduifb at gmx.de (Johannes Bauer) Date: Sun, 29 Dec 2019 23:02:48 +0100 Subject: '%Y' in strftime() vs. strptime() Message-ID: Hi list, I've just stumbled upon a strange phaenomenon and I'm wondering if it's a bug. Short and sweet: Python 3.7.3 (default, Oct 7 2019, 12:56:13) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from datetime import datetime as d >>> x = d(1, 1, 1) >>> x.strftime("%Y-%m-%d") '1-01-01' >>> d.strptime(x.strftime("%Y-%m-%d"), "%Y-%m-%d") Traceback (most recent call last): File "", line 1, in File "/usr/lib/python3.7/_strptime.py", line 577, in _strptime_datetime tt, fraction, gmtoff_fraction = _strptime(data_string, format) File "/usr/lib/python3.7/_strptime.py", line 359, in _strptime (data_string, format)) ValueError: time data '1-01-01' does not match format '%Y-%m-%d' >>> d.strptime("0001-01-01", "%Y-%m-%d") datetime.datetime(1, 1, 1, 0, 0) I.e. for years that are not 4 digits longs, strftime() produces no leading zeros for the '%Y' replacement, but strptime() requires leading zeros. Is this expected behavior? Shouldn't %Y be consistent across both? All the best, Johannes From eryksun at gmail.com Sun Dec 29 17:28:48 2019 From: eryksun at gmail.com (Eryk Sun) Date: Sun, 29 Dec 2019 16:28:48 -0600 Subject: '%Y' in strftime() vs. strptime() In-Reply-To: References: Message-ID: On 12/29/19, Johannes Bauer wrote: > >>>> x = d(1, 1, 1) > >>>> x.strftime("%Y-%m-%d") > '1-01-01' The default padding depends on the platform strftime. POSIX strftime [1] supports an extension of ISO C that allows specifying the "0" pad character and minimum field width, e.g. "%04Y-%02m-%02d". [1] https://pubs.opengroup.org/onlinepubs/9699919799/functions/strftime.html From safiq713 at gmail.com Mon Dec 30 00:20:54 2019 From: safiq713 at gmail.com (safiq713 at gmail.com) Date: Sun, 29 Dec 2019 21:20:54 -0800 (PST) Subject: name 'sys' is not defined Message-ID: <1325ad47-5b2d-4ee0-b3cd-1e7da33d9dc4@googlegroups.com> Deal all, Could you please help me how can I avoid this problem my Jupyter Notebook code was help pls # init a treemix analysis object with some param arguments tmx = ipa.treemix( data=data, imap=imap, minmap=minmap, seed=123456, root="Petronia_petronia", m=2, ) error NameError Traceback (most recent call last) in 6 seed=123456, 7 root="Petronia_petronia", ----> 8 m=2, 9 ) ~/opt/miniconda3/envs/py3/lib/python3.6/site-packages/ipyrad/analysis/treemix.py in __init__(self, data, name, workdir, imap, minmap, seed, quiet, raise_root_error, binary, *args, **kwargs) 118 119 # others --> 120 self.binary = os.path.join(sys.prefix, "bin", "treemix") 121 self.binary = (binary if binary else self.binary) 122 self.raise_root_error = raise_root_error NameError: name 'sys' is not defined From frank at chagford.com Mon Dec 30 00:49:43 2019 From: frank at chagford.com (Frank Millman) Date: Mon, 30 Dec 2019 07:49:43 +0200 Subject: name 'sys' is not defined In-Reply-To: <1325ad47-5b2d-4ee0-b3cd-1e7da33d9dc4@googlegroups.com> References: <1325ad47-5b2d-4ee0-b3cd-1e7da33d9dc4@googlegroups.com> Message-ID: On 2019-12-30 7:20 AM, safiq713 at gmail.com wrote: > Deal all, > Could you please help me how can I avoid this problem > my Jupyter Notebook code was > help pls > [snip] > > NameError: name 'sys' is not defined > I know nothing about Jupyter Notebook but somewhere, usually at the top, you have to add this line - import sys HTH Frank Millman From frank at chagford.com Mon Dec 30 00:49:43 2019 From: frank at chagford.com (Frank Millman) Date: Mon, 30 Dec 2019 07:49:43 +0200 Subject: name 'sys' is not defined In-Reply-To: <1325ad47-5b2d-4ee0-b3cd-1e7da33d9dc4@googlegroups.com> References: <1325ad47-5b2d-4ee0-b3cd-1e7da33d9dc4@googlegroups.com> Message-ID: On 2019-12-30 7:20 AM, safiq713 at gmail.com wrote: > Deal all, > Could you please help me how can I avoid this problem > my Jupyter Notebook code was > help pls > [snip] > > NameError: name 'sys' is not defined > I know nothing about Jupyter Notebook but somewhere, usually at the top, you have to add this line - import sys HTH Frank Millman From pieter-l at vanoostrum.org Mon Dec 30 10:04:46 2019 From: pieter-l at vanoostrum.org (Pieter van Oostrum) Date: Mon, 30 Dec 2019 16:04:46 +0100 Subject: name 'sys' is not defined References: <1325ad47-5b2d-4ee0-b3cd-1e7da33d9dc4@googlegroups.com> Message-ID: safiq713 at gmail.com writes: > Deal all, > Could you please help me how can I avoid this problem > my Jupyter Notebook code was > help pls > > > > # init a treemix analysis object with some param arguments > tmx = ipa.treemix( > data=data, > imap=imap, > minmap=minmap, > seed=123456, > root="Petronia_petronia", > m=2, > ) > > error > > NameError Traceback (most recent call last) > in > 6 seed=123456, > 7 root="Petronia_petronia", > ----> 8 m=2, > 9 ) > > ~/opt/miniconda3/envs/py3/lib/python3.6/site-packages/ipyrad/analysis/treemix.py in __init__(self, data, name, workdir, imap, minmap, seed, quiet, raise_root_error, binary, *args, **kwargs) > 118 > 119 # others > --> 120 self.binary = os.path.join(sys.prefix, "bin", "treemix") > 121 self.binary = (binary if binary else self.binary) > 122 self.raise_root_error = raise_root_error > > NameError: name 'sys' is not defined As the name sys is used in the imported module, that module has to import sys. Importing it in the calling code doesn't help. So I would say this is a bug in the module. You should report the bug to its author. In the meantime you can correct your own copy at ~/opt/miniconda3/envs/py3/lib/python3.6/site-packages/ipyrad/analysis/treemix.py by adding import sys for example around line 10 -- Pieter van Oostrum www: http://pieter.vanoostrum.org/ PGP key: [8DAE142BE17999C4] From barry at barrys-emacs.org Mon Dec 30 09:47:41 2019 From: barry at barrys-emacs.org (Barry Scott) Date: Mon, 30 Dec 2019 14:47:41 +0000 Subject: Friday Finking: Source code organisation In-Reply-To: References: Message-ID: > On 28 Dec 2019, at 22:49, Chris Angelico wrote: > > On Sun, Dec 29, 2019 at 9:37 AM DL Neil via Python-list > wrote: >> >> Is it helpful to, and thus, do you have a style/convention for ordering >> the methods within each class in your code? >> >> A major difference however, is that if our mythical collection of >> module-functions has an internal-reference, eg b() requires a(), then >> function a() MUST exist, ie be defined, 'before' function b(). Whereas a >> class's methods may be defined in any (complete) sequence. >> >> So, do you have an orderly method [hah!] for presenting/locating >> class-methods (and module-functions) within your code? >> >> - why bother, the editor does 'the heavy lifting' >> - dunders to the fore >> - alphanumeric sequence by name >> - order of appearance/use in 'mainline code' >> - as they sprang to mind during TDD-creation >> - most-used first, least-used last >> - my code 'at the top', their stuff later... >> - names of Monty Python characters by TV appearance date >> or, >> - some combination of ideas >> and, >> - how do you vary the above when dependencies intrude? >> > > "Define before use" is a broad principle that I try to follow, even > when the code itself doesn't mandate this. This generally means that > "if name is main" is the very last thing in the file, and if there's a > main() function or equivalent, that's usually just before that. Any > metaprogramming goes right at the top; sometimes this is mandated (if > I write a decorator function, it has to be above the functions it's > decorating), but even if it's not, metaprogramming goes before the > mainline. "define before use" is basically email top-posting for code isn't it? It means that the first things that you read in a module are the least interesting. I prefer to follow the a top-down design approach. Start with what is important and put what that depends on further down the file/class. With classes, __init__, __new__ then the public API and last the supporting code. Barry > > Other than that, I don't have any consistent logic other than a loose > idea of trying to keep related things together. > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > From rosuav at gmail.com Mon Dec 30 10:35:16 2019 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 31 Dec 2019 02:35:16 +1100 Subject: Friday Finking: Source code organisation In-Reply-To: References: Message-ID: On Tue, Dec 31, 2019 at 1:47 AM Barry Scott wrote: > > > > > On 28 Dec 2019, at 22:49, Chris Angelico wrote: > > > > On Sun, Dec 29, 2019 at 9:37 AM DL Neil via Python-list > > wrote: > >> > >> Is it helpful to, and thus, do you have a style/convention for ordering > >> the methods within each class in your code? > >> > >> A major difference however, is that if our mythical collection of > >> module-functions has an internal-reference, eg b() requires a(), then > >> function a() MUST exist, ie be defined, 'before' function b(). Whereas a > >> class's methods may be defined in any (complete) sequence. > >> > >> So, do you have an orderly method [hah!] for presenting/locating > >> class-methods (and module-functions) within your code? > >> > >> - why bother, the editor does 'the heavy lifting' > >> - dunders to the fore > >> - alphanumeric sequence by name > >> - order of appearance/use in 'mainline code' > >> - as they sprang to mind during TDD-creation > >> - most-used first, least-used last > >> - my code 'at the top', their stuff later... > >> - names of Monty Python characters by TV appearance date > >> or, > >> - some combination of ideas > >> and, > >> - how do you vary the above when dependencies intrude? > >> > > > > "Define before use" is a broad principle that I try to follow, even > > when the code itself doesn't mandate this. This generally means that > > "if name is main" is the very last thing in the file, and if there's a > > main() function or equivalent, that's usually just before that. Any > > metaprogramming goes right at the top; sometimes this is mandated (if > > I write a decorator function, it has to be above the functions it's > > decorating), but even if it's not, metaprogramming goes before the > > mainline. > > > > "define before use" is basically email top-posting for code isn't it? Kinda the opposite - it means that you get the context before you get something that refers to it. ChrisA From rosuav at gmail.com Mon Dec 30 11:21:53 2019 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 31 Dec 2019 03:21:53 +1100 Subject: Friday Finking: Source code organisation In-Reply-To: <6F82BBA4-6239-4654-B1A8-139740C3C01C@barrys-emacs.org> References: <6F82BBA4-6239-4654-B1A8-139740C3C01C@barrys-emacs.org> Message-ID: On Tue, Dec 31, 2019 at 3:08 AM Barry Scott wrote: > > > > > On 30 Dec 2019, at 15:35, Chris Angelico wrote: > > > > On Tue, Dec 31, 2019 at 1:47 AM Barry Scott wrote: > >> > >> > >> > >>> On 28 Dec 2019, at 22:49, Chris Angelico wrote: > >>> > >>> On Sun, Dec 29, 2019 at 9:37 AM DL Neil via Python-list > >>> wrote: > >>>> > >>>> Is it helpful to, and thus, do you have a style/convention for ordering > >>>> the methods within each class in your code? > >>>> > >>>> A major difference however, is that if our mythical collection of > >>>> module-functions has an internal-reference, eg b() requires a(), then > >>>> function a() MUST exist, ie be defined, 'before' function b(). Whereas a > >>>> class's methods may be defined in any (complete) sequence. > >>>> > >>>> So, do you have an orderly method [hah!] for presenting/locating > >>>> class-methods (and module-functions) within your code? > >>>> > >>>> - why bother, the editor does 'the heavy lifting' > >>>> - dunders to the fore > >>>> - alphanumeric sequence by name > >>>> - order of appearance/use in 'mainline code' > >>>> - as they sprang to mind during TDD-creation > >>>> - most-used first, least-used last > >>>> - my code 'at the top', their stuff later... > >>>> - names of Monty Python characters by TV appearance date > >>>> or, > >>>> - some combination of ideas > >>>> and, > >>>> - how do you vary the above when dependencies intrude? > >>>> > >>> > >>> "Define before use" is a broad principle that I try to follow, even > >>> when the code itself doesn't mandate this. This generally means that > >>> "if name is main" is the very last thing in the file, and if there's a > >>> main() function or equivalent, that's usually just before that. Any > >>> metaprogramming goes right at the top; sometimes this is mandated (if > >>> I write a decorator function, it has to be above the functions it's > >>> decorating), but even if it's not, metaprogramming goes before the > >>> mainline. > >> > >> > >> > >> "define before use" is basically email top-posting for code isn't it? > > > > Kinda the opposite - it means that you get the context before you get > > something that refers to it. > > But its not context its the low level detail that I should be able to ignore > when trying to understand the goals of the code. I'll only need to know > about the lower level detail if I have a need to change the code. > > Another analogy would ne how a journalist writes a story. You can take > a well written article and stop reading at any point, but still know what > the story is about. The more paragraphs you read the more informed > you are about the story. (Editors exploit this property of a story to shorten > it if it does not fit on the page). > Sure, but that's not what top-posting in emails is like. With a top-posted reply, truncation means you lack context; with the journalistic style of "front-load the important info", truncation means you lack detail. So in coding, I would say the journalistic style corresponds to a broad notion of "first docstring, then main function, then support code", which is an absolutely legit way to do things (but not my preferred way). ChrisA From barry at barrys-emacs.org Mon Dec 30 11:08:15 2019 From: barry at barrys-emacs.org (Barry Scott) Date: Mon, 30 Dec 2019 16:08:15 +0000 Subject: Friday Finking: Source code organisation In-Reply-To: References: Message-ID: <6F82BBA4-6239-4654-B1A8-139740C3C01C@barrys-emacs.org> > On 30 Dec 2019, at 15:35, Chris Angelico wrote: > > On Tue, Dec 31, 2019 at 1:47 AM Barry Scott wrote: >> >> >> >>> On 28 Dec 2019, at 22:49, Chris Angelico wrote: >>> >>> On Sun, Dec 29, 2019 at 9:37 AM DL Neil via Python-list >>> wrote: >>>> >>>> Is it helpful to, and thus, do you have a style/convention for ordering >>>> the methods within each class in your code? >>>> >>>> A major difference however, is that if our mythical collection of >>>> module-functions has an internal-reference, eg b() requires a(), then >>>> function a() MUST exist, ie be defined, 'before' function b(). Whereas a >>>> class's methods may be defined in any (complete) sequence. >>>> >>>> So, do you have an orderly method [hah!] for presenting/locating >>>> class-methods (and module-functions) within your code? >>>> >>>> - why bother, the editor does 'the heavy lifting' >>>> - dunders to the fore >>>> - alphanumeric sequence by name >>>> - order of appearance/use in 'mainline code' >>>> - as they sprang to mind during TDD-creation >>>> - most-used first, least-used last >>>> - my code 'at the top', their stuff later... >>>> - names of Monty Python characters by TV appearance date >>>> or, >>>> - some combination of ideas >>>> and, >>>> - how do you vary the above when dependencies intrude? >>>> >>> >>> "Define before use" is a broad principle that I try to follow, even >>> when the code itself doesn't mandate this. This generally means that >>> "if name is main" is the very last thing in the file, and if there's a >>> main() function or equivalent, that's usually just before that. Any >>> metaprogramming goes right at the top; sometimes this is mandated (if >>> I write a decorator function, it has to be above the functions it's >>> decorating), but even if it's not, metaprogramming goes before the >>> mainline. >> >> >> >> "define before use" is basically email top-posting for code isn't it? > > Kinda the opposite - it means that you get the context before you get > something that refers to it. But its not context its the low level detail that I should be able to ignore when trying to understand the goals of the code. I'll only need to know about the lower level detail if I have a need to change the code. Another analogy would ne how a journalist writes a story. You can take a well written article and stop reading at any point, but still know what the story is about. The more paragraphs you read the more informed you are about the story. (Editors exploit this property of a story to shorten it if it does not fit on the page). Barry > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > From amitsis at gmail.com Mon Dec 30 18:33:52 2019 From: amitsis at gmail.com (amitsis at gmail.com) Date: Mon, 30 Dec 2019 15:33:52 -0800 (PST) Subject: Segmentation violation with python installation under Tumbleweed Message-ID: Hello, My full Python installation doesn't seem to work due to a segmentation violation issue with libc-2.30: Dec 30 23:54:14 craylinux1 Kernel: Python [8467]: Segfault at 100013483 ip 00007f6594c896b6 sp 00007ffe1570d188 error 4 in libc-2.30.so [7f6594c10000 + 14d000] 30.12. 23:54:14 craylinux1 kernel: Code: 0f 1f 40 00 66 0f ef c0 66 0f ef c9 66 0f ef d2 66 0f ef db 48 89 f8 48 89 f9 48 81 e1 ff 0f 00 00 48 81 f9 cf 0f 00 00 77 6a 0f 6f 20 66 0f 74 e0 66 0f d7 d4 85 d2 74 04 0f bc c2 c3 48 83 I think this is just a bug of mismatched binaries and libraries in my Python installation. I want to reinstall the full Python installation including the libraries now, but how can you do it with Zypper to get a clean Python environment again? Any help with the detailed command lines would be greatly appreciated. Regards, Andreas From greg.ewing at canterbury.ac.nz Tue Dec 31 00:29:41 2019 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 31 Dec 2019 18:29:41 +1300 Subject: Friday Finking: Source code organisation In-Reply-To: References: Message-ID: On 31/12/19 3:47 am, Barry Scott wrote: > "define before use" is basically email top-posting for code isn't it? > > It means that the first things that you read in a module are the least interesting. That's not a big problem for top-level code, since you can easily scroll down to the bottom of the file and work backwards. But it's not so easy for classes if you have more than one class in a file. That might be part of the reason I do things the opposite way in classes. -- Greg