Is it possible to deliver different source distributions for different Python versions?
Dylan Evans
dylanlee.evans95 at gmail.com
Mon Apr 6 07:23:28 EDT 2015
> I would like to distribute a python package with different code for
> Python 2.* than for Python 3.*. (Mostly this is because of different
> unicode string handling).
> There is nothing in to setuptools or PyPi that directly supports
> this scenario.
The 2to3 utility will handle the unicode differences. You can set the keyword argument use_2to3 to True in setup.py dependent on the version number like so:
====================
extra = {}
if sys.version_info >= (3, ):
extra['use_2to3'] = True
====================
and then put **extra in setup.py. This will run 2to3 on the code when it installs it. I don’t know whether this is good style but it does work. I got it from the feedparser source code.
--
Dylan Evans
On 6 April 2015 at 11:05:47, python-list-request at python.org (python-list-request at python.org) wrote:
Send Python-list mailing list submissions to
python-list at python.org
To subscribe or unsubscribe via the World Wide Web, visit
https://mail.python.org/mailman/listinfo/python-list
or, via email, send a message with subject or body 'help' to
python-list-request at python.org
You can reach the person managing the list at
python-list-owner at python.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Python-list digest..."
Today's Topics:
1. Permission denied when opening a file that was created
concurrently by os.rename (Windows) (Alexey Izbyshev)
2. Re: Permission denied when opening a file that was created
concurrently by os.rename (Windows) (Chris Angelico)
3. Re: Permission denied when opening a file that was created
concurrently by os.rename (Windows) (Alexey Izbyshev)
4. Re: Permission denied when opening a file that was created
concurrently by os.rename (Windows) (Dave Angel)
5. Help with pipes, buffering and pseudoterminals (Daniel Ellis)
6. Re: Permission denied when opening a file that was created
concurrently by os.rename (Windows) (Terry Reedy)
7. Is it possible to deliver different source distributions for
different Python versions? (Dave Hein)
8. ANN: polynice 0.7 - a nice(1) like utility for throttling
processes (garabik-news-2005-05 at kassiopeia.juls.savba.sk)
9. Re: Help with pipes, buffering and pseudoterminals (Nobody)
10. Re: Help with pipes, buffering and pseudoterminals
(Cameron Simpson)
11. Re: Is it possible to deliver different source distributions
for different Python versions? (Steven D'Aprano)
12. XML Parsing (Sepideh Ghanavati)
13. Re: XML Parsing (Ben Finney)
14. Re: Is it possible to deliver different source distributions
for different Python versions? (Mark Lawrence)
15. Re: Is it possible to deliver different source distributions
for different Python versions? (Stefan Behnel)
16. Re: XML Parsing (Stefan Behnel)
17. Re: OOP for System Administrator (pankaj sharma)
18. implementing pyshark (Michael S.)
Hello!
I've hit a strange problem that I reduced to the following test case:
* Run several python processes in parallel that spin in the following
loop:
while True:
if os.path.isfile(fname):
with open(fname, 'rb') as f:
f.read()
break
* Then, run another process that creates a temporary file and then
renames it to the name than other processes are expecting
* Now, some of the reading processes occasionally fail with "Permission
denied" OSError
I was able to reproduce it on two Windows 7 64-bit machines. It seems
when the file appears on the filesystem it is still unavailable to
reading, but I have no idea how it can happen. Both source and
destination files are in the same directory, and the destination doesn't
exist before calling os.rename. Everything I could find indicates that
os.rename should be atomic under this conditions even on Windows, so
nobody should be able to observe the destination in unaccessible state.
I know that I can workaround this problem by removing useless
os.path.isfile() check and wrapping open() with try-except, but I'd like
to know the root cause of the problem. Please share you thoughts.
The test case is attached, the main file is test.bat. Python is
expected to be in PATH. Stderr of readers is redirected to *.log. You
may need to run several times to hit the issue.
Alexey Izbyshev,
research assistant,
ISP RAS
On Mon, Apr 6, 2015 at 3:45 AM, Alexey Izbyshev <izbyshev at ispras.ru> wrote:
> The test case is attached, the main file is test.bat. Python is expected to
> be in PATH. Stderr of readers is redirected to *.log. You may need to run
> several times to hit the issue.
You have an interesting-looking problem, but the attachment didn't
arrive. Is it short enough to include in-line as text in your email?
ChrisA
On 2015-04-05 20:45, Alexey Izbyshev wrote:
> Hello!
>
> I've hit a strange problem that I reduced to the following test case:
> * Run several python processes in parallel that spin in the following
> loop:
> while True:
> if os.path.isfile(fname):
> with open(fname, 'rb') as f:
> f.read()
> break
> * Then, run another process that creates a temporary file and then
> renames it to the name than other processes are expecting
> * Now, some of the reading processes occasionally fail with
> "Permission denied" OSError
>
> I was able to reproduce it on two Windows 7 64-bit machines. It seems
> when the file appears on the filesystem it is still unavailable to
> reading, but I have no idea how it can happen. Both source and
> destination files are in the same directory, and the destination
> doesn't exist before calling os.rename. Everything I could find
> indicates that os.rename should be atomic under this conditions even
> on Windows, so nobody should be able to observe the destination in
> unaccessible state.
>
> I know that I can workaround this problem by removing useless
> os.path.isfile() check and wrapping open() with try-except, but I'd
> like to know the root cause of the problem. Please share you thoughts.
>
> The test case is attached, the main file is test.bat. Python is
> expected to be in PATH. Stderr of readers is redirected to *.log. You
> may need to run several times to hit the issue.
>
> Alexey Izbyshev,
> research assistant,
> ISP RAS
The attachment was rejected because of .bat files, so here is the code:
test.bat:
======================
@echo off
del test.txt
del *.log
for /L %%i in (1, 1, 10) do (
start /b run-with-log.bat %%i
)
pythonw test-rename.py
======================
run-with-log.bat:
======================
@echo off
pythonw test-read.py %1 >nul 2>%1.log
exit
======================
On 04/05/2015 01:45 PM, Alexey Izbyshev wrote:
> Hello!
>
> I've hit a strange problem that I reduced to the following test case:
> * Run several python processes in parallel that spin in the following loop:
> while True:
> if os.path.isfile(fname):
> with open(fname, 'rb') as f:
> f.read()
> break
> * Then, run another process that creates a temporary file and then
> renames it to the name than other processes are expecting
> * Now, some of the reading processes occasionally fail with "Permission
> denied" OSError
>
> I was able to reproduce it on two Windows 7 64-bit machines. It seems
> when the file appears on the filesystem it is still unavailable to
> reading, but I have no idea how it can happen. Both source and
> destination files are in the same directory, and the destination doesn't
> exist before calling os.rename. Everything I could find indicates that
> os.rename should be atomic under this conditions even on Windows, so
> nobody should be able to observe the destination in unaccessible state.
>
> I know that I can workaround this problem by removing useless
> os.path.isfile() check and wrapping open() with try-except, but I'd like
> to know the root cause of the problem. Please share you thoughts.
>
> The test case is attached, the main file is test.bat. Python is expected
> to be in PATH. Stderr of readers is redirected to *.log. You may need to
> run several times to hit the issue.
>
> Alexey Izbyshev,
> research assistant,
> ISP RAS
>
The attachment is missing; please just include it inline, after
reducing it to a reasonably minimal sample.
My guess is that the process that does the os.rename is not closing the
original file before renaming it. So even though the rename is atomic,
the file is still locked by the first process.
--
DaveA
I have a small little tool I'd like to make. It essentially takes piped input, modifies the text in some way, and immediately prints the output. The problem I'm having is that any output I pipe to the program seems to be buffered, removing the desired effect.
>From what I understand, I need to somehow have the input be retrieved via a pseudoterminal. The problem that I'm having is that most examples on the internet seem to assume I would like to launch a program in a forked pty process, which doesn't really fit my use case.
I've tried a number of things, but I seem to be unable to get even a basic understanding of how to use the pty module. Here's a piece of code I whipped up just to try to get a feel for what is going on when I use pty.fork, but it doesn't seem to do what I think it should:
import pty
import os
import sys
pid, fd = pty.fork()
print pid, fd
sys.stdout.flush()
os.read(fd, 1024)
This only seems to print from the parent process. I read that I need to do the os.read call for the fork to happen. I've also tried printing *after* the os.read call.
I realize this does very little to solve my overall goal, but I figure understanding what is going on is probably a worthwhile first step.
On 4/5/2015 1:45 PM, Alexey Izbyshev wrote:
> Hello!
>
> I've hit a strange problem that I reduced to the following test case:
> * Run several python processes in parallel that spin in the following loop:
> while True:
> if os.path.isfile(fname):
> with open(fname, 'rb') as f:
> f.read()
> break
Besides anything else, I would consider adding a minimal sleep in the loop.
> * Then, run another process that creates a temporary file and then
> renames it to the name than other processes are expecting
> * Now, some of the reading processes occasionally fail with "Permission
> denied" OSError
--
Terry Jan Reedy
I would like to distribute a python package with different code for
Python 2.* than for Python 3.*. (Mostly this is because of different
unicode string handling).
There is nothing in to setuptools or PyPi that directly supports
this scenario.
But perhaps there could be some script run at install time that moves
the correct source code to the right location? In other works, if I
included both source code versions in the distribution (in a src2 and
a src3 subdirectory) then a function invoked at install time could
detect the python version and copy the appropriate source code to the
right location.
Is that at all possible? Is there some install time hook that lets me
supply custom installation code?
--
Dave Hein
polynice is a nice(1)-like command line utility for unix systems to
throttle long running processes beyond what can be achieved by nice(1),
by repeatedly suspending and resuming the process.
It is written for python3, though there is some python2.7 compatibility.
Author:
Radovan Garabík
URL:
http://kassiopeia.juls.savba.sk/~garabik/software/polynice.html
License:
GPL (v2)
Notable changes:
* This is the first public release under the name `polynice' - the
utility has been renamed from `verynice' due to name clash with an
existing software.
* polynice can now emulate timeout(1) - terminate process after given
time has passed
* MacOS X improvements
* better python2 compatiblity
--
-----------------------------------------------------------
| Radovan Garabík http://kassiopeia.juls.savba.sk/~garabik/ |
| __..--^^^--..__ garabik @ kassiopeia.juls.savba.sk |
-----------------------------------------------------------
Antivirus alert: file .signature infected by signature virus.
Hi! I'm a signature virus! Copy me into your signature file to help me spread!
On Sun, 05 Apr 2015 12:20:48 -0700, Daniel Ellis wrote:
> This only seems to print from the parent process. I read that I need to
> do the os.read call for the fork to happen. I've also tried printing
> *after* the os.read call.
The child process has its std{in,out,err} attached to the newly-created
pty, so that's where the output from the child's "print" goes.
You'll see that output if the parent prints the string returned from the
os.read() call.
On 05Apr2015 12:20, Daniel Ellis <ellisd23 at gmail.com> wrote:
>I have a small little tool I'd like to make. It essentially takes piped input, modifies the text in some way, and immediately prints the output. The problem I'm having is that any output I pipe to the program seems to be buffered, removing the desired effect.
That depends on the upstream program; it does the buffering. The pipe itself
presents received data downstream immediately.
However, as you've seen, almost every program buffers its standard output if
the output is not a tty; this is automatic in the stdio C library and results
in more fficient use of I/O.
>From what I understand, I need to somehow have the input be retrieved via a pseudoterminal.
This is a rather gross hack, though sometimes all you can do. While some
programs have an option to force unbuffered output, most do not. Attaching
their output to a pty is one way to encourage them to at least line buffer
their output.
However, you should bear in mind that the reason that programs line buffer to a
terminal is that they presume they are in an interactive situation with a
person watching. The program _may_ act differently in other ways as well, such
as asking question it might not otherwise ask in "batch" mode (where it might
cautiously not ask and presume "no").
Also, output sent through a pty is subject to the line discipline in the
terminal; temrinals are funny things with much historical behaviour. At the
least you pobably want your pty in "raw" mode to avoid all sorts of stuff that
can be done to your data.
>The problem that I'm having is that most examples on the internet seem to assume I would like to launch a program in a forked pty process, which doesn't really fit my use case.
Indeed not, but not to worry. You don't need to fork.
>I've tried a number of things, but I seem to be unable to get even a basic understanding of how to use the pty module.
Have you every used a pty from C? Do you know how ptys work? (master side,
slave side, etc).
>Here's a piece of code I whipped up just to try to get a feel for what is going on when I use pty.fork, but it doesn't seem to do what I think it should:
>
> import pty
> import os
> import sys
>
> pid, fd = pty.fork()
> print pid, fd
> sys.stdout.flush()
> os.read(fd, 1024)
>
>This only seems to print from the parent process.
The documentation for pty.fork says:
Return value is (pid, fd). Note that the child gets pid 0, and the fd is invalid.
So the child cannot used "fd". It further says that the child has its stdin and
stdout attached to the pty, and that the pty is the child's controlling
terminal (this means it is affected by things like "typing" ^C at the pty,
etc).
>I read that I need to do the os.read call for the fork to happen. I've also
tried printing *after* the os.read call.
Don't try to adapt fork-based tutorials to your needs. Understand ptys directly
first.
>I realize this does very little to solve my overall goal, but I figure understanding what is going on is probably a worthwhile first step.
What you probably want to use is pty.openpty() instead. No fork. You will get
back file descriptors for the master and slave sides of the pty. Then you can
use these with the subprocess module to connect your input program. Or,
guessing from your opening sentence, you can write a wrapper script whose whole
purpose is to run a program on a pty.
Regarding terminology: a pseudoterminal (pty) is a device that looks like a
traditional serial terminal. All terminal emulators like xterm use one, and so
do other programs presenting a terminal session such as the sshd process
handling an interactive remote login.
When you call pty.openpty() you are handed two file descriptors: one for the
master side of the pty and one for the slave side. The slave side is the side
that looks like a terminal, and is what a typical use would connect a child
process to. The master side is the other side of the pty. When a program writes
to the "slave" side, the output is available for read on the master side, much
like a pipe. When a program writes to the master side, the output is available
for read on the slave side, _as_ _if_ _typed_ at the terminal.
A pty is not necessarily going to solve your problem unless you can get your
input via the pty. From the sounds of it you're in this situation:
command-generating-output | your-program
such that your input is attached to a pipe, and because
"command-generating-output" is attached to a pipe it is block buffering its
output, hence your problem.
You can't undo that situation after the fact.
To solve your problem via a pty you need to contrive to set up
"command-generating-output" already attached to a pty. One way to do that is
for "your-program" to open a pty and itself invoke "command-generating-output"
with its output via the pty, which is why so many tutorials suppose a "fork"
situation.
One typical away to do that is to pass the "command-generating-output" command
name and args to your program, eg:
your-program command-generating-output [args...]
Then your main program can gather that up:
import sys
command_generating_output = sys.argv[1:]
Then you can call pty.openpty(), and then use the slave file descriptor with
subprocess.Popen to invoke command_generating_output. Thus the generating
command will be talking to you via a pty instead of a pipe.
Cheers,
Cameron Simpson <cs at zip.com.au>
It is interesting to think of the great blaze of heaven that we winnow
down to animal shapes and kitchen tools. - Don DeLillo
On Mon, 6 Apr 2015 06:38 am, Dave Hein wrote:
> I would like to distribute a python package with different code for
> Python 2.* than for Python 3.*. (Mostly this is because of different
> unicode string handling).
>
> There is nothing in to setuptools or PyPi that directly supports
> this scenario.
>
> But perhaps there could be some script run at install time that moves
> the correct source code to the right location? In other works, if I
> included both source code versions in the distribution (in a src2 and
> a src3 subdirectory) then a function invoked at install time could
> detect the python version and copy the appropriate source code to the
> right location.
>
> Is that at all possible? Is there some install time hook that lets me
> supply custom installation code?
I'm not aware of any standard solution to that, but I'm not a setuptools
expert. setup.py files are Python code, so you can put any code you like in
them. But, as far as I am concerned, having the installer pick and choose
what source files to include is not a good solution. Instead, you should
pick one of these two alternatives:
(1) Supply a separate package for 2.x and 3.x, each with their own
installer. The installer confirms that it is running under the correct
version of Python, and just installs.
(2) Or supply a single package with a single installer that works under both
2.x and 3.x. (This is my preference.)
One way of handling the second case is to only support 3.3 or better: that
way, you can still use u"..." for Unicode strings. Hardly anyone used 3.1
or 3.2, and 3.0 is no longer supported, so it is quite reasonable to insist
people upgrade to 3.3.
Another way to handle the second case is to use conditional imports:
# mymodule2
mystring = u"äπЖ☃" # Also works in 3.3 or better.
# mymodule3
mystring = "äπЖ☃"
# main application
from __future__ import print_function
from future_builtins import *
if sys.version < '3':
import mymodule2 as mymodule
else:
import mymodule3 as mymodule
print(mymodule.mystring)
--
Steven
Hi,
I know basic of python and I have an xml file created from csv which has three attributes "category", "definition" and "definition description". I want to parse through xml file and identify actors, constraints, principal from the text. However, I am not sure what is the best way to go. Any suggestion?
Sepideh Ghanavati <sepideh.gh at gmail.com> writes:
> I know basic of python and I have an xml file created from csv
What XML schema defines the document's format?
Without knowing the schema, parsing will be unreliable.
What created the document? Why is it relevant that the document was
“created from CSV”?
> which has three attributes "category", "definition" and "definition
> description".
What do you mean by “attributes”?
In Python, an attribute has a specific meaning.
In XML, an attribute has a rather different meaning.
Neither of those meanings seems to apply to “the XML document has three
attributes”. XML documents don't have attributes; differnt XML elements
in a document have different attributes.
> I want to parse through xml file and identify actors, constraints,
> principal from the text.
How are those defined in the document's schema?
> However, I am not sure what is the best way to go. Any suggestion?
You should:
* Learn some more about XML
<URL:http://www.xmlobjective.com/the-basic-principles-of-xml/>.
* Learn exactly what formal document schema defines the document
<URL:https://en.wikipedia.org/wiki/XML_schema>. If the document isn't
accompanied by a specification of exactly what its schema is, you're
going to have a difficult time.
--
\ “If I melt dry ice, can I swim without getting wet?” —Steven |
`\ Wright |
_o__) |
Ben Finney
On 05/04/2015 21:38, Dave Hein wrote:
> I would like to distribute a python package with different code for
> Python 2.* than for Python 3.*. (Mostly this is because of different
> unicode string handling).
>
> There is nothing in to setuptools or PyPi that directly supports
> this scenario.
>
> But perhaps there could be some script run at install time that moves
> the correct source code to the right location? In other works, if I
> included both source code versions in the distribution (in a src2 and
> a src3 subdirectory) then a function invoked at install time could
> detect the python version and copy the appropriate source code to the
> right location.
>
> Is that at all possible? Is there some install time hook that lets me
> supply custom installation code?
>
> --
> Dave Hein
>
I can't help directly but have you looked here
https://packaging.python.org/en/latest/ ?
--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.
Mark Lawrence
Dave Hein schrieb am 05.04.2015 um 22:38:
> I would like to distribute a python package with different code for
> Python 2.* than for Python 3.*. (Mostly this is because of different
> unicode string handling).
>
> There is nothing in to setuptools or PyPi that directly supports
> this scenario.
>
> But perhaps there could be some script run at install time that moves
> the correct source code to the right location? In other works, if I
> included both source code versions in the distribution (in a src2 and
> a src3 subdirectory) then a function invoked at install time could
> detect the python version and copy the appropriate source code to the
> right location.
>
> Is that at all possible? Is there some install time hook that lets me
> supply custom installation code?
Sure. You can simply change the directory in which distutils looks for your
Python code:
https://docs.python.org/2/distutils/setupscript.html#listing-whole-packages
However, in general, you shouldn't be doing this. It's usually easier
(definitely in the long-term) to keep your sources cross-Py2.x/3.x
compatible, maybe with the help of tools like "six" or "python-future",
than to try to keep separate source trees in sync.
http://python-future.org/
Stefan
Sepideh Ghanavati schrieb am 06.04.2015 um 04:26:
> I know basic of python and I have an xml file created from csv which has
> three attributes "category", "definition" and "definition description".
> I want to parse through xml file and identify actors, constraints,
> principal from the text. However, I am not sure what is the best way to
> go. Any suggestion?
If it's really generated from a CSV file, you could also parse that instead:
https://docs.python.org/3/library/csv.html
Admittedly, CSV files are simple, but they also have major problems,
especially when it comes to detecting their character encoding and their
specific format (tab/comma/semicolon/space/whatever separated, with or
without escaping, quoted values, ...). Meaning, you can easily end up
reading nonsense from the file instead of the content that was originally
put into it.
So, if you want to parse from XML instead, use ElementTree:
https://docs.python.org/3/library/xml.etree.elementtree.html
Stefan
Yes Michael, your guess is right. I once was a Java Programmer :)
Thanks for the advice and link.
Regards,
Hi everyone. while trying to implement pyshark, i am getting this error:
import pyshark
capture = pyshark.LiveCapture(interface='wlan0')
capture.sniff(timeout=50)
capture
i have tried also to run it through interpreter yet, i got this error:
import pyshark
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-1-e46ddf42872e> in <module>()
----> 1 import pyshark
/usr/local/lib/python2.7/dist-packages/pyshark-0.3.4-py2.7.egg/pyshark/__init__.py
in <module>()
----> 1 from pyshark.capture.live_capture import LiveCapture
2 from pyshark.capture.file_capture import FileCapture
3 from pyshark.capture.remote_capture import RemoteCapture
4 from pyshark.capture.inmem_capture import InMemCapture
/usr/local/lib/python2.7/dist-packages/pyshark-0.3.4-py2.7.egg/pyshark/capture/live_capture.py
in <module>()
----> 1 from pyshark.capture.capture import Capture
2 from pyshark.tshark.tshark import get_tshark_interfaces
3
4
5 class LiveCapture(Capture):
/usr/local/lib/python2.7/dist-packages/pyshark-0.3.4-py2.7.egg/pyshark/capture/capture.py
in <module>()
1 from __future__ import unicode_literals
2 import os
----> 3 import logbook
4 import sys
5
/usr/local/lib/python2.7/dist-packages/logbook/__init__.py in <module>()
12
13
---> 14 from logbook.base import LogRecord, Logger, LoggerGroup,
NestedSetup, \
15 Processor, Flags, get_level_name, lookup_level,
dispatch_record, \
16 CRITICAL, ERROR, WARNING, NOTICE, INFO, DEBUG, NOTSET, \
/usr/local/lib/python2.7/dist-packages/logbook/base.py in <module>()
16 from datetime import datetime
17 from logbook import helpers
---> 18 from logbook.concurrency import thread_get_name,
thread_get_ident, greenlet_get_ident
19
20 from logbook.helpers import to_safe_json, parse_iso8601,
cached_property, \
/usr/local/lib/python2.7/dist-packages/logbook/concurrency.py in <module>()
29
30 if has_gevent:
---> 31 from gevent._threading import (Lock as ThreadLock,
32 RLock as ThreadRLock,
33 get_ident as thread_get_ident,
ImportError: No module named _threading
i tried importing threading first and importing pyshark, but the answer
was the same.
any suggestions?
--
https://mail.python.org/mailman/listinfo/python-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20150406/375a9311/attachment.html>
More information about the Python-list
mailing list