From deeppunster at gmail.com  Thu Feb  1 22:48:20 2018
From: deeppunster at gmail.com (Travis Risner)
Date: Thu, 01 Feb 2018 22:48:20 -0500
Subject: [CentralOH] Pythology: Python Security Best Practices (February
 2)
In-Reply-To: <CAGCtaoMgZgoda+W6BwHuLH_mVEj88YySDaY+tPgZ+xjL9chR3Q@mail.gmail.com>
References: <CAJzarMOssh_V_Z6XOgVzv2bS-k6SMf-7dcHbSymO3wtgThJyEA@mail.gmail.com>
 <CAGCtaoMgZgoda+W6BwHuLH_mVEj88YySDaY+tPgZ+xjL9chR3Q@mail.gmail.com>
Message-ID: <5A73DF84.9020005@gmail.com>

Hi Catherine,

Thanks for the generous offer.  Circumstances prevented me from signing
up for this event this year so I can't take you up on your offer this time.

I plan to go to the PyCon in Cleveland so maybe I will see you there?
If not, then we can still meet again at the PyOhio.

Cheers!
Travis

Catherine Devlin wrote:
> Hi, Columbus folks!  If anybody can make it to Brookville (exit 21), I
> can offer you an eco-friendly Prius ride the rest of the way.
> 
> - Catherine
> 
> On Wed, Jan 17, 2018 at 3:05 PM, Eric Floehr <eric at intellovations.com
> <mailto:eric at intellovations.com>> wrote:
> 
>     It?s almost time for IndyPy?s next Pythology One-Day Conference!
> 
[snip]
> 
> -- 
> - Catherine
> http://catherinedevlin.blogspot.com
> 
> _______________________________________________
> CentralOH mailing list
> CentralOH at python.org
> https://mail.python.org/mailman/listinfo/centraloh

-- 
Sent by Travis Risner

From jep200404 at columbus.rr.com  Fri Feb  2 09:43:23 2018
From: jep200404 at columbus.rr.com (jep200404 at columbus.rr.com)
Date: Fri, 2 Feb 2018 09:43:23 -0500
Subject: [CentralOH] =?utf-8?q?2018-01-29_=E6=9C=83=E8=AD=B0_Scribbles_?=
 =?utf-8?b?76SY5pu4L+aDoeaWhz86IEZpbGVzIGZvciBNaWNoYWVsIEhhbmRsZXIncyAi?=
 =?utf-8?q?Rudimentary_Web_Monitoring=22_Presentation?=
In-Reply-To: <20180130224926.3bce6ff2.jep200404@columbus.rr.com>
References: <20180130224926.3bce6ff2.jep200404@columbus.rr.com>
Message-ID: <20180202094323.20b55653.jep200404@columbus.rr.com>

On Tue, 30 Jan 2018 22:49:26 -0500, jep200404 at columbus.rr.com wrote:

> there were two presentations
> 
>     the main one was Michael Handler's:
> 
>         Michael Handler will present a Python script that uses
>         Selenium and Firefox to implement some rudimentary web
>         scraping to monitor a web application. This script is used in
>         conjunction with an open source monitoring application called
>         Zabbix to monitor the operation of a web application we host
>         for another business unit of his employer's parent company.

The files for that presentation are at
https://github.com/cohpy/20180129-rudimentary-web-monitoring

From jep200404 at columbus.rr.com  Fri Feb  2 22:45:59 2018
From: jep200404 at columbus.rr.com (jep200404 at columbus.rr.com)
Date: Fri, 2 Feb 2018 22:45:59 -0500
Subject: [CentralOH] =?utf-8?q?2018-02-01_=E9=81=93=E5=A0=B4_Scribbles_?=
 =?utf-8?b?76SY5pu4L+aDoeaWhz8gcGZkYTsgdHdvIHNjb29wczsgZGphbmdvb3JtOyBw?=
 =?utf-8?q?ip_wheel=3B_bash=3B_grep=3B_find=3B_xargs=3B_parsing_HTML?=
Message-ID: <20180202224559.29393045.jep200404@columbus.rr.com>

    PfDA
        Python for Data Analysis, 2nd Edition - O'Reilly Media
        http://shop.oreilly.com/product/0636920050896.do

        old version
        Python for Data Analysis
        http://shop.oreilly.com/product/0636920023784.do

Two Scoops of Django 1.11: Best Practices for the Django Web Framework
https://www.twoscoopspress.com/products/two-scoops-of-django-1-11

django
    800 pound gorilla
    djangoorm
        migration support is fantastic

Cost of Living Comparison Between Columbus, OH and Cleveland, OH
https://www.numbeo.com/cost-of-living/compare_cities.jsp?country1=United+States&city1=Columbus%2C+OH&country2=United+States&city2=Cleveland%2C+OH

Cost of Living Comparison Between Columbus, OH and Cincinnati, OH
https://www.numbeo.com/cost-of-living/compare_cities.jsp?country1=United+States&city1=Columbus%2C+OH&country2=United+States&city2=Cincinnati%2C+OH

Cost of Living Comparison Between Columbus, OH and Raleigh, NC
https://www.numbeo.com/cost-of-living/compare_cities.jsp?country1=United+States&city1=Columbus%2C+OH&country2=United+States&city2=Raleigh%2C+NC

Cost of Living Comparison Between Columbus, OH and Ann Arbor, MI
https://www.numbeo.com/cost-of-living/compare_cities.jsp?country1=United+States&city1=Columbus%2C+OH&country2=United+States&city2=Ann%20Arbor%2C+MI

Cost of Living Comparison Between Columbus, OH and Des Moines, IA
https://www.numbeo.com/cost-of-living/compare_cities.jsp?country1=United+States&city1=Columbus%2C+OH&country2=United+States&city2=Des%20Moines%2C+IA

Cost of Living Comparison Between Columbus, OH and Manchester, NH
https://www.numbeo.com/cost-of-living/compare_cities.jsp?country1=United+States&city1=Columbus%2C+OH&country2=United+States&city2=Manchester%2C+NH

Cost of Living Comparison Between Columbus, OH and Santa Cruz, CA
https://www.numbeo.com/cost-of-living/compare_cities.jsp?country1=United+States&city1=Columbus%2C+OH&country2=United+States&city2=Santa%20Cruz%2C+CA

Cost of Living Comparison Between Columbus, OH and Boston, MA
https://www.numbeo.com/cost-of-living/compare_cities.jsp?country1=United+States&city1=Columbus%2C+OH&country2=United+States&city2=Boston%2C+MA

wheel

https://duckduckgo.com/html/?q=microsoft+bash+shell

wp:grep
wp:find (Unix)
wp:xargs

wp: prefix means Wikipedia
To get good answers, consider following the advice in the links below.
http://catb.org/~esr/faqs/smart-questions.html
http://web.archive.org/web/20090627155454/www.greenend.org.uk/rjk/2000/06/14/quoting.html

grep -r ManagementUtility .
grep -lr ManagementUtility .
grep -lr ManagementUtility . | grep -v '[.]pyc$'
vi -O `grep -lr ManagementUtility . | grep -v '[.]pyc$'`
find . -type f -exec grep -l 'ManagementUtility' {} \;
find . -type f -print0 | xargs -0 grep -l 'ManagementUtility'
find . -type f -print0 | xargs -0 -n 1 grep -l 'ManagementUtility'
grep -rl 'ManagementUtility' .
find . -type f -exec grep -l 'ManagementUtility' {} \;
find . -type f -name '*.py' -print0 | xargs -0 -n 1 grep -l 'ManagementUtility'

mysql versus mariadb
    mariadb is the new mysql

wp:HL7

wifi was lousy tonight

Excellent No-Charge Linux Books ? Learn About Linux ? For Beginners
https://www.linuxlinks.com/excellent-no-charge-books-learn-about-linux/

Why you should use named pipes on Linux
https://www.networkworld.com/article/3251853/linux/why-use-named-pipes-on-linux.html

Parsing HTML with Python
https://opensource.com/article/18/1/parsing-html-python

Very Heavy Ion Collider at DISCERN
https://xkcd.com/

How I coined the term 'open source'
https://opensource.com/article/18/2/coining-term-open-source-software

10 tips for making the Django Admin more secure
https://opensource.com/article/18/1/10-tips-making-django-admin-more-secure

wp:The Four Feathers (2002 film)

From jep200404 at columbus.rr.com  Sun Feb 11 14:50:54 2018
From: jep200404 at columbus.rr.com (jep200404 at columbus.rr.com)
Date: Sun, 11 Feb 2018 14:50:54 -0500
Subject: [CentralOH] =?utf-8?q?2018-02-08_=E6=9C=83=E8=AD=B0_Scribbles_?=
 =?utf-8?b?76SY5pu4L+aDoeaWhz86IFJlZmFjdG9yaW5nOiBFQUZQIHYgTEJZTDsgRFJZ?=
 =?utf-8?q?=3B_PEP8=3B_Python_v_Java?=
Message-ID: <20180211145054.58198d7e.jep200404@columbus.rr.com>

a great time was had refactoring

    Compare the left and right sides:
    https://github.com/jocassid/JohnsUsefulPythonCode/pull/4/files?diff=split

    Study the differences of each commit for better understanding.
    One can start at the beginning of the changes:
    https://github.com/jocassid/JohnsUsefulPythonCode/pull/4/commits/c9f3f47
    Also read the commit comments.

wp:multiset
https://github.com/isometric/BucketCompressionTrick

wp: prefix means Wikipedia
To get good answers, consider following the advice in the links below.
http://catb.org/~esr/faqs/smart-questions.html
http://web.archive.org/web/20090627155454/www.greenend.org.uk/rjk/2000/06/14/quoting.html

Top 7 open source project management tools for agile teams
https://opensource.com/article/18/2/agile-project-management-tools

Thanks to Bruce Eckel for releasing TIJ4 for free
and his kind comment about Python.

Thinking in Java, 4th edition

    https://archive.org/details/TIJ4CcR1

        Programming is about managing complexity: the complexity
        of the problem you want to solve, laid upon the
        complexity of the machine in which it is solved. Because
        of this complexity, most of our programming projects
        fail. And yet, of all the programming languages of which
        I am aware, almost none have gone all out and decided
        that their main design goal would be to conquer the
        complexity of developing and maintaining programs. [1]

        [1] However, I believe that the Python language comes
        closest to doing exactly that. See www.Python.org.

learn java programming with 9 excellent free open-source books
https://www.ossblog.org/learn-java-programming-with-excellent-open-source-books/

java programming language profile
https://www.ossblog.org/java-programming-language-profile/

    Introduction to Programming Using Java, Seventh Edition
    http://math.hws.edu/javanotes/

    Think Java ? How to Think Like a Computer Scientist
    http://greenteapress.com/wp/think-java/

    Building Back-End Web Apps with Java, JPA and JSF
    http://web-engineering.info/tech/JavaJpaJsf/book/

man byteorder
    https://docs.python.org/3.5/library/socket.html#socket.ntohl
    https://docs.python.org/3.5/library/socket.html#socket.ntohs
    https://docs.python.org/3.5/library/socket.html#socket.htonl
    https://docs.python.org/3.5/library/socket.html#socket.htons
man endian
    the following do not match
        https://docs.python.org/3/library/array.html
        https://docs.python.org/3/library/array.html#array.array.byteswap

N.Korea Behind Massive Bitcoin Heists
http://english.chosun.com/site/data/html_dir/2018/02/06/2018020600817.html

(Paid) Jupyter Notebooks hosted on cloud
https://cocalc.com/
    sage
    julia
    (and python of course)

Power(Shell) to the people
https://opensource.com/article/18/2/powershell-people

One million Linux and open-source software classes taken
Getting an IT job today starts with taking Linux and open-source classes.
http://www.zdnet.com/article/one-million-linux-and-open-source-software-classes-taken/

How to Create, Revert and Delete KVM Virtual machine (domain) snapshot with virsh command
https://www.linuxtechi.com/create-revert-delete-kvm-virtual-machine-snapshot-virsh-command/

celery and rabbitmq

agnes meyer driscoll
grace murray hopper

elite dangerous star map
edsm.net
https://github.com/jocassid

https://github.com/jocassid/EDMarketConnector

clara peller
where's the beef

spyder is IDE that comes with anaconda
    like rstudio but less slick

sphynx is great for generating many formats from restructured text
wp:Sphinx_(documentation_generator)
wp:reStructuredText

11 Excellent Free Scorewriters ? Compose, arrange, print, and publish music
https://www.linuxlinks.com/excellent-scorewriters-compose-great-music/
wp:G?del, Escher, Bach

diff -u: Detainting the Kernel
http://www.linuxjournal.com/content/diff-u-detainting-kernel

Have mp3 patents expired in USA?

3 steps to reduce a project's failure rate
https://opensource.com/open-organization/18/2/constructing-project-requirements

From jep200404 at columbus.rr.com  Mon Feb 12 20:51:46 2018
From: jep200404 at columbus.rr.com (jep200404 at columbus.rr.com)
Date: Mon, 12 Feb 2018 20:51:46 -0500
Subject: [CentralOH] =?utf-8?q?2018-02-08_=E6=9C=83=E8=AD=B0_Scribbles_?=
 =?utf-8?b?76SY5pu4L+aDoeaWhz86IFJlZmFjdG9yaW5nOiBFQUZQIHYgTEJZTA==?=
In-Reply-To: <20180211145054.58198d7e.jep200404@columbus.rr.com>
References: <20180211145054.58198d7e.jep200404@columbus.rr.com>
Message-ID: <20180212205146.37ee4aa1.jep200404@columbus.rr.com>

On Sun, 11 Feb 2018 14:50:54 -0500, jep200404 at columbus.rr.com wrote:

> a great time was had refactoring

EAFP v LBYL was an important part of the refactoring.
EAFP v LBYL was covered earlier:

    https://github.com/james-prior/cohpy/blob/master/20170424-cohpy-lbyl-v-eafp.ipynb
    http://nbviewer.jupyter.org/github/james-prior/cohpy/blob/master/20170424-cohpy-lbyl-v-eafp.ipynb

From pcarswell.1 at gmail.com  Wed Feb 14 11:15:39 2018
From: pcarswell.1 at gmail.com (Peter Carswell)
Date: Wed, 14 Feb 2018 11:15:39 -0500
Subject: [CentralOH] 
	=?utf-8?q?2018-02-08_=E6=9C=83=E8=AD=B0_Scribbles_?=
	=?utf-8?b?76SY5pu4L+aDoeaWhz86IFJlZmFjdG9yaW5nOiBFQUZQIHYgTEJZTA==?=
In-Reply-To: <20180212205146.37ee4aa1.jep200404@columbus.rr.com>
References: <20180211145054.58198d7e.jep200404@columbus.rr.com>
 <20180212205146.37ee4aa1.jep200404@columbus.rr.com>
Message-ID: <CAAZoxjqjJ_9rZCWGZ8x7fx9NMjD6UvR24EOPMa4Wdp7SN_vHvg@mail.gmail.com>

hello Jim,

I was contacted by a teacher in a local high school with a problem. One of
the students taking the python class only has a chromebook. There are many
online resources for doing this. Do you have any suggestions? As to which
one you find superior? Thanks.

pete

On Mon, Feb 12, 2018 at 8:51 PM, <jep200404 at columbus.rr.com> wrote:

> On Sun, 11 Feb 2018 14:50:54 -0500, jep200404 at columbus.rr.com wrote:
>
> > a great time was had refactoring
>
> EAFP v LBYL was an important part of the refactoring.
> EAFP v LBYL was covered earlier:
>
>     https://github.com/james-prior/cohpy/blob/master/
> 20170424-cohpy-lbyl-v-eafp.ipynb
>     http://nbviewer.jupyter.org/github/james-prior/cohpy/blob/
> master/20170424-cohpy-lbyl-v-eafp.ipynb
> _______________________________________________
> CentralOH mailing list
> CentralOH at python.org
> https://mail.python.org/mailman/listinfo/centraloh
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/centraloh/attachments/20180214/f87b8e60/attachment.html>

From damiencalloway at gmail.com  Thu Feb 15 19:13:23 2018
From: damiencalloway at gmail.com (Damien Calloway)
Date: Thu, 15 Feb 2018 19:13:23 -0500
Subject: [CentralOH] 
	=?utf-8?q?2018-02-08_=E6=9C=83=E8=AD=B0_Scribbles_?=
	=?utf-8?b?76SY5pu4L+aDoeaWhz86IFJlZmFjdG9yaW5nOiBFQUZQIHYgTEJZTA==?=
In-Reply-To: <CAAZoxjqjJ_9rZCWGZ8x7fx9NMjD6UvR24EOPMa4Wdp7SN_vHvg@mail.gmail.com>
References: <20180211145054.58198d7e.jep200404@columbus.rr.com>
 <20180212205146.37ee4aa1.jep200404@columbus.rr.com>
 <CAAZoxjqjJ_9rZCWGZ8x7fx9NMjD6UvR24EOPMa4Wdp7SN_vHvg@mail.gmail.com>
Message-ID: <CAChJNcxmvZhf1Y2kVcJqJUoT-_8xxskfVUyfOoewHC8xLDDr5w@mail.gmail.com>

What is the nature of the course ? One solid option for a web-based
environment is http://jupyter.org/ - this is used very frequently at Python
meetings, dojo, etc.

How much collaboration does the course require ? And does any of the course
work require any kind of graphics programming (i.e. PyGame, etc.) ?

On Wednesday, February 14, 2018, Peter Carswell <pcarswell.1 at gmail.com>
wrote:

> hello Jim,
>
> I was contacted by a teacher in a local high school with a problem. One of
> the students taking the python class only has a chromebook. There are many
> online resources for doing this. Do you have any suggestions? As to which
> one you find superior? Thanks.
>
> pete
>
> On Mon, Feb 12, 2018 at 8:51 PM, <jep200404 at columbus.rr.com> wrote:
>
>> On Sun, 11 Feb 2018 14:50:54 -0500, jep200404 at columbus.rr.com wrote:
>>
>> > a great time was had refactoring
>>
>> EAFP v LBYL was an important part of the refactoring.
>> EAFP v LBYL was covered earlier:
>>
>>     https://github.com/james-prior/cohpy/blob/master/20170424-
>> cohpy-lbyl-v-eafp.ipynb
>>     http://nbviewer.jupyter.org/github/james-prior/cohpy/blob/ma
>> ster/20170424-cohpy-lbyl-v-eafp.ipynb
>> _______________________________________________
>> CentralOH mailing list
>> CentralOH at python.org
>> https://mail.python.org/mailman/listinfo/centraloh
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/centraloh/attachments/20180215/261c68cd/attachment.html>

From michaelburton.mjb at gmail.com  Fri Feb 16 12:35:36 2018
From: michaelburton.mjb at gmail.com (Michael Burton)
Date: Fri, 16 Feb 2018 12:35:36 -0500
Subject: [CentralOH] 2018-02-08 ?? Scribbles ??/???: Refactoring: EAFP v
 LBY
Message-ID: <CANnahyYe3CcY2sL6=ELGnM4wt7HneR+aK01Xm=1m+4PhKCzMQA@mail.gmail.com>

I am teaching Python via Jupyter notebooks in CoCalc (goto CoCalc.com).
Full collaboration in the cloud; completely agnostic as to the machine.
There is a slow free tier, but the upgrade only costs a few dollars per
month.  I don't have to worry about students installing, configuring, etc.

Mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/centraloh/attachments/20180216/62e9c1c6/attachment.html>

From jocassid at gmail.com  Fri Feb 16 15:39:11 2018
From: jocassid at gmail.com (John Cassidy)
Date: Fri, 16 Feb 2018 15:39:11 -0500
Subject: [CentralOH] 
	=?utf-8?q?2018-02-08_=E6=9C=83=E8=AD=B0_Scribbles_?=
	=?utf-8?b?76SY5pu4L+aDoeaWhz86IFJlZmFjdG9yaW5nOiBFQUZQIHYgTEJZTA==?=
In-Reply-To: <CAAZoxjqjJ_9rZCWGZ8x7fx9NMjD6UvR24EOPMa4Wdp7SN_vHvg@mail.gmail.com>
References: <20180211145054.58198d7e.jep200404@columbus.rr.com>
 <20180212205146.37ee4aa1.jep200404@columbus.rr.com>
 <CAAZoxjqjJ_9rZCWGZ8x7fx9NMjD6UvR24EOPMa4Wdp7SN_vHvg@mail.gmail.com>
Message-ID: <CAGt6GbdJp49TcsrCbC8bbwopi=SqGQc1R4t9JVwdOt8HRv_shg@mail.gmail.com>

Pete, pythonanywhere.com has an online Python development environment
(editor and terminal).  They have a freebie beginner level account.  Also
check out https://www.pythonanywhere.com/details/education

On Wed, Feb 14, 2018 at 11:15 AM, Peter Carswell <pcarswell.1 at gmail.com>
wrote:

> hello Jim,
>
> I was contacted by a teacher in a local high school with a problem. One of
> the students taking the python class only has a chromebook. There are many
> online resources for doing this. Do you have any suggestions? As to which
> one you find superior? Thanks.
>
> pete
>
> On Mon, Feb 12, 2018 at 8:51 PM, <jep200404 at columbus.rr.com> wrote:
>
>> On Sun, 11 Feb 2018 14:50:54 -0500, jep200404 at columbus.rr.com wrote:
>>
>> > a great time was had refactoring
>>
>> EAFP v LBYL was an important part of the refactoring.
>> EAFP v LBYL was covered earlier:
>>
>>     https://github.com/james-prior/cohpy/blob/master/20170424-
>> cohpy-lbyl-v-eafp.ipynb
>>     http://nbviewer.jupyter.org/github/james-prior/cohpy/blob/ma
>> ster/20170424-cohpy-lbyl-v-eafp.ipynb
>> _______________________________________________
>> CentralOH mailing list
>> CentralOH at python.org
>> https://mail.python.org/mailman/listinfo/centraloh
>>
>
>
> _______________________________________________
> CentralOH mailing list
> CentralOH at python.org
> https://mail.python.org/mailman/listinfo/centraloh
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/centraloh/attachments/20180216/e454df3c/attachment.html>

From eric at intellovations.com  Fri Feb 16 22:34:35 2018
From: eric at intellovations.com (Eric Floehr)
Date: Fri, 16 Feb 2018 22:34:35 -0500
Subject: [CentralOH] Popen file descriptor difference between Python 2.6 and
 2.7+ on Linux
Message-ID: <CAJzarMMSBicLqwiaULKupvYL5vHp7ptx6jMPjyaDEyJKEpsBiw@mail.gmail.com>

I have a really odd problem. I have some code that requires a Linux file
descriptor (/dev/fd/N) where N is some number. The following example code
demonstrates effectively what is being done.

In order to run the code create a file called "/tmp/test" with some text in
it.

The code works in Python 2.6, but in Python 2.7 or later, I get:

cat: /dev/fd/4: No such file or directory

Can anyone figure out why? And what needs to change for the following code
to work on 2.7 and later. Assume that I cannot change needing file
descriptors (/dev/fd/4 for example):

Here is the code:

-------------------------------------------------
from subprocess import Popen, PIPE

data = Popen(
    ["cat"],
    stdin=open('/tmp/test', 'rb'),
    stdout=PIPE,
    universal_newlines=True).stdout

fd_name = '/dev/fd/%d' % data.fileno()

fddata = Popen(
    ["cat",
    fd_name],
    stdout=PIPE,
    universal_newlines=True,).stdout

print(fddata.read())
-------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/centraloh/attachments/20180216/f0d222e3/attachment.html>

From nludban at columbus.rr.com  Fri Feb 16 23:32:41 2018
From: nludban at columbus.rr.com (Neil Ludban)
Date: Fri, 16 Feb 2018 23:32:41 -0500
Subject: [CentralOH] Popen file descriptor difference between Python 2.6
 and 2.7+ on Linux
In-Reply-To: <CAJzarMMSBicLqwiaULKupvYL5vHp7ptx6jMPjyaDEyJKEpsBiw@mail.gmail.com>
References: <CAJzarMMSBicLqwiaULKupvYL5vHp7ptx6jMPjyaDEyJKEpsBiw@mail.gmail.com>
Message-ID: <20180216233241.158f64c222273d749fe75ec7@columbus.rr.com>

On Fri, 16 Feb 2018 22:34:35 -0500
Eric Floehr <eric at intellovations.com> wrote:
> I have a really odd problem. I have some code that requires a Linux file
> descriptor (/dev/fd/N) where N is some number. The following example code
> demonstrates effectively what is being done.
> 
> In order to run the code create a file called "/tmp/test" with some text in
> it.
> 
> The code works in Python 2.6, but in Python 2.7 or later, I get:
> 
> cat: /dev/fd/4: No such file or directory

Smells like a back-porting of adding the O_CLOEXEC flag...

     Unless O_CLOEXEC flag was specified, the new descriptor is set to remain
     open across execve(2) system calls; see close(2), fcntl(2) and O_CLOEXEC
     description.

https://www.python.org/dev/peps/pep-0446/

From winningham at gmail.com  Sat Feb 17 00:38:50 2018
From: winningham at gmail.com (Thomas Winningham)
Date: Sat, 17 Feb 2018 00:38:50 -0500
Subject: [CentralOH] Popen file descriptor difference between Python 2.6
 and 2.7+ on Linux
In-Reply-To: <20180216233241.158f64c222273d749fe75ec7@columbus.rr.com>
References: <CAJzarMMSBicLqwiaULKupvYL5vHp7ptx6jMPjyaDEyJKEpsBiw@mail.gmail.com>
 <20180216233241.158f64c222273d749fe75ec7@columbus.rr.com>
Message-ID: <CAJShLf9eqTWJUSci3WziPwAk1eF3yeqcw6xNRDTTGXY8hO=wqg@mail.gmail.com>

I was replying on Twitter, but I suspect the behavior slowly became
undefined somewhere between 2.7 and 3 ... but after 3.2 i can do this and
it works:
------------------------
from subprocess import Popen, PIPE

data = Popen(
    ["cat"],
    stdin=open('/tmp/test', 'rb'),
    stdout=PIPE,
    stderr=PIPE,
    universal_newlines=True,
    shell=False).stdout

fd_name = '/dev/fd/%d' % data.fileno()

fddata = Popen(
    ["cat",fd_name],
    stdout=PIPE,
    universal_newlines=True,
    pass_fds=(data.fileno(),)).stdout

print(fddata.read())
------------------------
the "pass_fds" option was added in 3.2 per the subprocess documentation


On Fri, Feb 16, 2018 at 11:32 PM, Neil Ludban <nludban at columbus.rr.com>
wrote:

> On Fri, 16 Feb 2018 22:34:35 -0500
> Eric Floehr <eric at intellovations.com> wrote:
> > I have a really odd problem. I have some code that requires a Linux file
> > descriptor (/dev/fd/N) where N is some number. The following example code
> > demonstrates effectively what is being done.
> >
> > In order to run the code create a file called "/tmp/test" with some text
> in
> > it.
> >
> > The code works in Python 2.6, but in Python 2.7 or later, I get:
> >
> > cat: /dev/fd/4: No such file or directory
>
> Smells like a back-porting of adding the O_CLOEXEC flag...
>
>      Unless O_CLOEXEC flag was specified, the new descriptor is set to
> remain
>      open across execve(2) system calls; see close(2), fcntl(2) and
> O_CLOEXEC
>      description.
>
> https://www.python.org/dev/peps/pep-0446/
> _______________________________________________
> CentralOH mailing list
> CentralOH at python.org
> https://mail.python.org/mailman/listinfo/centraloh
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/centraloh/attachments/20180217/01016d88/attachment-0001.html>

From winningham at gmail.com  Sat Feb 17 00:39:40 2018
From: winningham at gmail.com (Thomas Winningham)
Date: Sat, 17 Feb 2018 00:39:40 -0500
Subject: [CentralOH] Popen file descriptor difference between Python 2.6
 and 2.7+ on Linux
In-Reply-To: <CAJShLf9eqTWJUSci3WziPwAk1eF3yeqcw6xNRDTTGXY8hO=wqg@mail.gmail.com>
References: <CAJzarMMSBicLqwiaULKupvYL5vHp7ptx6jMPjyaDEyJKEpsBiw@mail.gmail.com>
 <20180216233241.158f64c222273d749fe75ec7@columbus.rr.com>
 <CAJShLf9eqTWJUSci3WziPwAk1eF3yeqcw6xNRDTTGXY8hO=wqg@mail.gmail.com>
Message-ID: <CAJShLf_BUAGigV_LDKE7hwnR0Fp5EhZ4ZZB=3Ee57AnwABPdzA@mail.gmail.com>

oh and the shell=False and stderr = PIPE are not needed that was me just
trying to fudge things somehow and left those in there in that code...
really the pass_fds flag is the thing

On Sat, Feb 17, 2018 at 12:38 AM, Thomas Winningham <winningham at gmail.com>
wrote:

> I was replying on Twitter, but I suspect the behavior slowly became
> undefined somewhere between 2.7 and 3 ... but after 3.2 i can do this and
> it works:
> ------------------------
> from subprocess import Popen, PIPE
>
> data = Popen(
>     ["cat"],
>     stdin=open('/tmp/test', 'rb'),
>     stdout=PIPE,
>     stderr=PIPE,
>     universal_newlines=True,
>     shell=False).stdout
>
> fd_name = '/dev/fd/%d' % data.fileno()
>
> fddata = Popen(
>     ["cat",fd_name],
>     stdout=PIPE,
>     universal_newlines=True,
>     pass_fds=(data.fileno(),)).stdout
>
> print(fddata.read())
> ------------------------
> the "pass_fds" option was added in 3.2 per the subprocess documentation
>
>
> On Fri, Feb 16, 2018 at 11:32 PM, Neil Ludban <nludban at columbus.rr.com>
> wrote:
>
>> On Fri, 16 Feb 2018 22:34:35 -0500
>> Eric Floehr <eric at intellovations.com> wrote:
>> > I have a really odd problem. I have some code that requires a Linux file
>> > descriptor (/dev/fd/N) where N is some number. The following example
>> code
>> > demonstrates effectively what is being done.
>> >
>> > In order to run the code create a file called "/tmp/test" with some
>> text in
>> > it.
>> >
>> > The code works in Python 2.6, but in Python 2.7 or later, I get:
>> >
>> > cat: /dev/fd/4: No such file or directory
>>
>> Smells like a back-porting of adding the O_CLOEXEC flag...
>>
>>      Unless O_CLOEXEC flag was specified, the new descriptor is set to
>> remain
>>      open across execve(2) system calls; see close(2), fcntl(2) and
>> O_CLOEXEC
>>      description.
>>
>> https://www.python.org/dev/peps/pep-0446/
>> _______________________________________________
>> CentralOH mailing list
>> CentralOH at python.org
>> https://mail.python.org/mailman/listinfo/centraloh
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/centraloh/attachments/20180217/4de1eca0/attachment.html>

From eric at intellovations.com  Sat Feb 17 11:20:17 2018
From: eric at intellovations.com (Eric Floehr)
Date: Sat, 17 Feb 2018 11:20:17 -0500
Subject: [CentralOH] Popen file descriptor difference between Python 2.6
 and 2.7+ on Linux
In-Reply-To: <CAJShLf_BUAGigV_LDKE7hwnR0Fp5EhZ4ZZB=3Ee57AnwABPdzA@mail.gmail.com>
References: <CAJzarMMSBicLqwiaULKupvYL5vHp7ptx6jMPjyaDEyJKEpsBiw@mail.gmail.com>
 <20180216233241.158f64c222273d749fe75ec7@columbus.rr.com>
 <CAJShLf9eqTWJUSci3WziPwAk1eF3yeqcw6xNRDTTGXY8hO=wqg@mail.gmail.com>
 <CAJShLf_BUAGigV_LDKE7hwnR0Fp5EhZ4ZZB=3Ee57AnwABPdzA@mail.gmail.com>
Message-ID: <CAJzarMObD6nJPJ3fo13c+jEtagkySh0fXrm7YwhbX318dPqRQQ@mail.gmail.com>

Thank you Neil and Thomas, here is a summary of what you discovered:

There is a regression in Python 2.7 where file descriptors are not passed
to child processes in the subprocess module, whereas in Python 2.6 they
were.

In order to address this, they added a "pass_fds" parameter to Popen in
Python 3.2. However Python 2.7 does not have this parameter.

So, in order for this to work on Python 2.7, I resorted to using named
temporary files. This code won't work on Windows NT and higher (per Python
documentation, because named temporary files can't be opened more than once
on Windows NT and higher but can on UNIX-like systems).

Here is the summary of changes:

1. Wrote contents of first Popen's stdout to a NamedTemporaryFile
2. Passed the name of the NamedTemporaryFile to the second Popen
3. Added with blocks to ensure both Popen's get closed properly
4. Closed the NamedTemporaryFile after the second Popen finished, to delete
the temp file off disk.

Here is the code:

--------------------------------------------------
from subprocess import Popen, PIPE
import tempfile

with Popen(
        ["cat"],
        stdin=open('/tmp/test', 'rb'),
        stdout=PIPE,
        universal_newlines=True).stdout as data:
    tmpfile = tempfile.NamedTemporaryFile()
    tmpfile.write(data.read())
    tmpfile.flush()

with Popen(
        ["cat",
        tmpfile.name],
        stdout=PIPE,
        universal_newlines=True,).stdout as fddata:
    print(fddata.read())

tmpfile.close()
--------------------------------------------------


This works on 2.6, 2.7, and 3.5. Thanks again for all of your help!
Eric



On Sat, Feb 17, 2018 at 12:39 AM, Thomas Winningham <winningham at gmail.com>
wrote:

> oh and the shell=False and stderr = PIPE are not needed that was me just
> trying to fudge things somehow and left those in there in that code...
> really the pass_fds flag is the thing
>
> On Sat, Feb 17, 2018 at 12:38 AM, Thomas Winningham <winningham at gmail.com>
> wrote:
>
>> I was replying on Twitter, but I suspect the behavior slowly became
>> undefined somewhere between 2.7 and 3 ... but after 3.2 i can do this and
>> it works:
>> ------------------------
>> from subprocess import Popen, PIPE
>>
>> data = Popen(
>>     ["cat"],
>>     stdin=open('/tmp/test', 'rb'),
>>     stdout=PIPE,
>>     stderr=PIPE,
>>     universal_newlines=True,
>>     shell=False).stdout
>>
>> fd_name = '/dev/fd/%d' % data.fileno()
>>
>> fddata = Popen(
>>     ["cat",fd_name],
>>     stdout=PIPE,
>>     universal_newlines=True,
>>     pass_fds=(data.fileno(),)).stdout
>>
>> print(fddata.read())
>> ------------------------
>> the "pass_fds" option was added in 3.2 per the subprocess documentation
>>
>>
>> On Fri, Feb 16, 2018 at 11:32 PM, Neil Ludban <nludban at columbus.rr.com>
>> wrote:
>>
>>> On Fri, 16 Feb 2018 22:34:35 -0500
>>> Eric Floehr <eric at intellovations.com> wrote:
>>> > I have a really odd problem. I have some code that requires a Linux
>>> file
>>> > descriptor (/dev/fd/N) where N is some number. The following example
>>> code
>>> > demonstrates effectively what is being done.
>>> >
>>> > In order to run the code create a file called "/tmp/test" with some
>>> text in
>>> > it.
>>> >
>>> > The code works in Python 2.6, but in Python 2.7 or later, I get:
>>> >
>>> > cat: /dev/fd/4: No such file or directory
>>>
>>> Smells like a back-porting of adding the O_CLOEXEC flag...
>>>
>>>      Unless O_CLOEXEC flag was specified, the new descriptor is set to
>>> remain
>>>      open across execve(2) system calls; see close(2), fcntl(2) and
>>> O_CLOEXEC
>>>      description.
>>>
>>> https://www.python.org/dev/peps/pep-0446/
>>> _______________________________________________
>>> CentralOH mailing list
>>> CentralOH at python.org
>>> https://mail.python.org/mailman/listinfo/centraloh
>>>
>>
>>
>
> _______________________________________________
> CentralOH mailing list
> CentralOH at python.org
> https://mail.python.org/mailman/listinfo/centraloh
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/centraloh/attachments/20180217/2827a87e/attachment-0001.html>

From winningham at gmail.com  Sat Feb 17 18:42:08 2018
From: winningham at gmail.com (Thomas Winningham)
Date: Sat, 17 Feb 2018 18:42:08 -0500
Subject: [CentralOH] Popen file descriptor difference between Python 2.6
 and 2.7+ on Linux
In-Reply-To: <CAJzarMObD6nJPJ3fo13c+jEtagkySh0fXrm7YwhbX318dPqRQQ@mail.gmail.com>
References: <CAJzarMMSBicLqwiaULKupvYL5vHp7ptx6jMPjyaDEyJKEpsBiw@mail.gmail.com>
 <20180216233241.158f64c222273d749fe75ec7@columbus.rr.com>
 <CAJShLf9eqTWJUSci3WziPwAk1eF3yeqcw6xNRDTTGXY8hO=wqg@mail.gmail.com>
 <CAJShLf_BUAGigV_LDKE7hwnR0Fp5EhZ4ZZB=3Ee57AnwABPdzA@mail.gmail.com>
 <CAJzarMObD6nJPJ3fo13c+jEtagkySh0fXrm7YwhbX318dPqRQQ@mail.gmail.com>
Message-ID: <CAJShLf-=DaTtMs+o+cannpGUwWUydnwskpyE1LcQi7UgkG5_AA@mail.gmail.com>

Well, sucks you had to go through this rewrite. I think it shows that all
code has a certain level of implicit functionality whose interpretation
varies between stakeholders over time. So while it took a little digging, I
think it also shows that everything is eventually knowable if needed.

"Explicit is better than implicit" although the original code seemed rather
explicit its own right, this new code is certainly more explicit. If IO
performance becomes an issue with the tempfile, maybe named pipes could be
used? I learned a lot about per-process file descriptors though, and
reinforces how recently I learned that *nix opens a file based on
descriptor and not by name whereas like you say Windows can only have a
file opened by one process at a time because the file name is the
descriptor in a sense.

There's always a way to further over engineer this probably too, I'm
thinking with sockets or even a message queue :P


On Sat, Feb 17, 2018 at 11:20 AM, Eric Floehr <eric at intellovations.com>
wrote:

> Thank you Neil and Thomas, here is a summary of what you discovered:
>
> There is a regression in Python 2.7 where file descriptors are not passed
> to child processes in the subprocess module, whereas in Python 2.6 they
> were.
>
> In order to address this, they added a "pass_fds" parameter to Popen in
> Python 3.2. However Python 2.7 does not have this parameter.
>
> So, in order for this to work on Python 2.7, I resorted to using named
> temporary files. This code won't work on Windows NT and higher (per Python
> documentation, because named temporary files can't be opened more than once
> on Windows NT and higher but can on UNIX-like systems).
>
> Here is the summary of changes:
>
> 1. Wrote contents of first Popen's stdout to a NamedTemporaryFile
> 2. Passed the name of the NamedTemporaryFile to the second Popen
> 3. Added with blocks to ensure both Popen's get closed properly
> 4. Closed the NamedTemporaryFile after the second Popen finished, to
> delete the temp file off disk.
>
> Here is the code:
>
> --------------------------------------------------
> from subprocess import Popen, PIPE
> import tempfile
>
> with Popen(
>         ["cat"],
>         stdin=open('/tmp/test', 'rb'),
>         stdout=PIPE,
>         universal_newlines=True).stdout as data:
>     tmpfile = tempfile.NamedTemporaryFile()
>     tmpfile.write(data.read())
>     tmpfile.flush()
>
> with Popen(
>         ["cat",
>         tmpfile.name],
>         stdout=PIPE,
>         universal_newlines=True,).stdout as fddata:
>     print(fddata.read())
>
> tmpfile.close()
> --------------------------------------------------
>
>
> This works on 2.6, 2.7, and 3.5. Thanks again for all of your help!
> Eric
>
>
>
> On Sat, Feb 17, 2018 at 12:39 AM, Thomas Winningham <winningham at gmail.com>
> wrote:
>
>> oh and the shell=False and stderr = PIPE are not needed that was me just
>> trying to fudge things somehow and left those in there in that code...
>> really the pass_fds flag is the thing
>>
>> On Sat, Feb 17, 2018 at 12:38 AM, Thomas Winningham <winningham at gmail.com
>> > wrote:
>>
>>> I was replying on Twitter, but I suspect the behavior slowly became
>>> undefined somewhere between 2.7 and 3 ... but after 3.2 i can do this and
>>> it works:
>>> ------------------------
>>> from subprocess import Popen, PIPE
>>>
>>> data = Popen(
>>>     ["cat"],
>>>     stdin=open('/tmp/test', 'rb'),
>>>     stdout=PIPE,
>>>     stderr=PIPE,
>>>     universal_newlines=True,
>>>     shell=False).stdout
>>>
>>> fd_name = '/dev/fd/%d' % data.fileno()
>>>
>>> fddata = Popen(
>>>     ["cat",fd_name],
>>>     stdout=PIPE,
>>>     universal_newlines=True,
>>>     pass_fds=(data.fileno(),)).stdout
>>>
>>> print(fddata.read())
>>> ------------------------
>>> the "pass_fds" option was added in 3.2 per the subprocess documentation
>>>
>>>
>>> On Fri, Feb 16, 2018 at 11:32 PM, Neil Ludban <nludban at columbus.rr.com>
>>> wrote:
>>>
>>>> On Fri, 16 Feb 2018 22:34:35 -0500
>>>> Eric Floehr <eric at intellovations.com> wrote:
>>>> > I have a really odd problem. I have some code that requires a Linux
>>>> file
>>>> > descriptor (/dev/fd/N) where N is some number. The following example
>>>> code
>>>> > demonstrates effectively what is being done.
>>>> >
>>>> > In order to run the code create a file called "/tmp/test" with some
>>>> text in
>>>> > it.
>>>> >
>>>> > The code works in Python 2.6, but in Python 2.7 or later, I get:
>>>> >
>>>> > cat: /dev/fd/4: No such file or directory
>>>>
>>>> Smells like a back-porting of adding the O_CLOEXEC flag...
>>>>
>>>>      Unless O_CLOEXEC flag was specified, the new descriptor is set to
>>>> remain
>>>>      open across execve(2) system calls; see close(2), fcntl(2) and
>>>> O_CLOEXEC
>>>>      description.
>>>>
>>>> https://www.python.org/dev/peps/pep-0446/
>>>> _______________________________________________
>>>> CentralOH mailing list
>>>> CentralOH at python.org
>>>> https://mail.python.org/mailman/listinfo/centraloh
>>>>
>>>
>>>
>>
>> _______________________________________________
>> CentralOH mailing list
>> CentralOH at python.org
>> https://mail.python.org/mailman/listinfo/centraloh
>>
>>
>
> _______________________________________________
> CentralOH mailing list
> CentralOH at python.org
> https://mail.python.org/mailman/listinfo/centraloh
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/centraloh/attachments/20180217/e50c9099/attachment.html>

From eric at intellovations.com  Mon Feb 19 15:55:00 2018
From: eric at intellovations.com (Eric Floehr)
Date: Mon, 19 Feb 2018 15:55:00 -0500
Subject: [CentralOH] Popen file descriptor difference between Python 2.6
 and 2.7+ on Linux
In-Reply-To: <CAJShLf-=DaTtMs+o+cannpGUwWUydnwskpyE1LcQi7UgkG5_AA@mail.gmail.com>
References: <CAJzarMMSBicLqwiaULKupvYL5vHp7ptx6jMPjyaDEyJKEpsBiw@mail.gmail.com>
 <20180216233241.158f64c222273d749fe75ec7@columbus.rr.com>
 <CAJShLf9eqTWJUSci3WziPwAk1eF3yeqcw6xNRDTTGXY8hO=wqg@mail.gmail.com>
 <CAJShLf_BUAGigV_LDKE7hwnR0Fp5EhZ4ZZB=3Ee57AnwABPdzA@mail.gmail.com>
 <CAJzarMObD6nJPJ3fo13c+jEtagkySh0fXrm7YwhbX318dPqRQQ@mail.gmail.com>
 <CAJShLf-=DaTtMs+o+cannpGUwWUydnwskpyE1LcQi7UgkG5_AA@mail.gmail.com>
Message-ID: <CAJzarMNvKKDVOJAHTgg41nO0UjbZxpuix0_vk3h5=kaQ9XNppw@mail.gmail.com>

Good ideas... the actual application is gunzipping two ASCII files
(gzipp'ed for disk space consideration) and diff'ing them.

These files are added to but not just appended to, so we need to find the
changes between two files. We do that, rather than processing the whole
file each time because these are large files with lots of row and parsing
and determining if a row's data exists or has changed would be hugely
expensive. And the system diff program is the best for that.





On Sat, Feb 17, 2018 at 6:42 PM, Thomas Winningham <winningham at gmail.com>
wrote:

> Well, sucks you had to go through this rewrite. I think it shows that all
> code has a certain level of implicit functionality whose interpretation
> varies between stakeholders over time. So while it took a little digging, I
> think it also shows that everything is eventually knowable if needed.
>
> "Explicit is better than implicit" although the original code seemed
> rather explicit its own right, this new code is certainly more explicit. If
> IO performance becomes an issue with the tempfile, maybe named pipes could
> be used? I learned a lot about per-process file descriptors though, and
> reinforces how recently I learned that *nix opens a file based on
> descriptor and not by name whereas like you say Windows can only have a
> file opened by one process at a time because the file name is the
> descriptor in a sense.
>
> There's always a way to further over engineer this probably too, I'm
> thinking with sockets or even a message queue :P
>
>
> On Sat, Feb 17, 2018 at 11:20 AM, Eric Floehr <eric at intellovations.com>
> wrote:
>
>> Thank you Neil and Thomas, here is a summary of what you discovered:
>>
>> There is a regression in Python 2.7 where file descriptors are not passed
>> to child processes in the subprocess module, whereas in Python 2.6 they
>> were.
>>
>> In order to address this, they added a "pass_fds" parameter to Popen in
>> Python 3.2. However Python 2.7 does not have this parameter.
>>
>> So, in order for this to work on Python 2.7, I resorted to using named
>> temporary files. This code won't work on Windows NT and higher (per Python
>> documentation, because named temporary files can't be opened more than once
>> on Windows NT and higher but can on UNIX-like systems).
>>
>> Here is the summary of changes:
>>
>> 1. Wrote contents of first Popen's stdout to a NamedTemporaryFile
>> 2. Passed the name of the NamedTemporaryFile to the second Popen
>> 3. Added with blocks to ensure both Popen's get closed properly
>> 4. Closed the NamedTemporaryFile after the second Popen finished, to
>> delete the temp file off disk.
>>
>> Here is the code:
>>
>> --------------------------------------------------
>> from subprocess import Popen, PIPE
>> import tempfile
>>
>> with Popen(
>>         ["cat"],
>>         stdin=open('/tmp/test', 'rb'),
>>         stdout=PIPE,
>>         universal_newlines=True).stdout as data:
>>     tmpfile = tempfile.NamedTemporaryFile()
>>     tmpfile.write(data.read())
>>     tmpfile.flush()
>>
>> with Popen(
>>         ["cat",
>>         tmpfile.name],
>>         stdout=PIPE,
>>         universal_newlines=True,).stdout as fddata:
>>     print(fddata.read())
>>
>> tmpfile.close()
>> --------------------------------------------------
>>
>>
>> This works on 2.6, 2.7, and 3.5. Thanks again for all of your help!
>> Eric
>>
>>
>>
>> On Sat, Feb 17, 2018 at 12:39 AM, Thomas Winningham <winningham at gmail.com
>> > wrote:
>>
>>> oh and the shell=False and stderr = PIPE are not needed that was me just
>>> trying to fudge things somehow and left those in there in that code...
>>> really the pass_fds flag is the thing
>>>
>>> On Sat, Feb 17, 2018 at 12:38 AM, Thomas Winningham <
>>> winningham at gmail.com> wrote:
>>>
>>>> I was replying on Twitter, but I suspect the behavior slowly became
>>>> undefined somewhere between 2.7 and 3 ... but after 3.2 i can do this and
>>>> it works:
>>>> ------------------------
>>>> from subprocess import Popen, PIPE
>>>>
>>>> data = Popen(
>>>>     ["cat"],
>>>>     stdin=open('/tmp/test', 'rb'),
>>>>     stdout=PIPE,
>>>>     stderr=PIPE,
>>>>     universal_newlines=True,
>>>>     shell=False).stdout
>>>>
>>>> fd_name = '/dev/fd/%d' % data.fileno()
>>>>
>>>> fddata = Popen(
>>>>     ["cat",fd_name],
>>>>     stdout=PIPE,
>>>>     universal_newlines=True,
>>>>     pass_fds=(data.fileno(),)).stdout
>>>>
>>>> print(fddata.read())
>>>> ------------------------
>>>> the "pass_fds" option was added in 3.2 per the subprocess documentation
>>>>
>>>>
>>>> On Fri, Feb 16, 2018 at 11:32 PM, Neil Ludban <nludban at columbus.rr.com>
>>>> wrote:
>>>>
>>>>> On Fri, 16 Feb 2018 22:34:35 -0500
>>>>> Eric Floehr <eric at intellovations.com> wrote:
>>>>> > I have a really odd problem. I have some code that requires a Linux
>>>>> file
>>>>> > descriptor (/dev/fd/N) where N is some number. The following example
>>>>> code
>>>>> > demonstrates effectively what is being done.
>>>>> >
>>>>> > In order to run the code create a file called "/tmp/test" with some
>>>>> text in
>>>>> > it.
>>>>> >
>>>>> > The code works in Python 2.6, but in Python 2.7 or later, I get:
>>>>> >
>>>>> > cat: /dev/fd/4: No such file or directory
>>>>>
>>>>> Smells like a back-porting of adding the O_CLOEXEC flag...
>>>>>
>>>>>      Unless O_CLOEXEC flag was specified, the new descriptor is set to
>>>>> remain
>>>>>      open across execve(2) system calls; see close(2), fcntl(2) and
>>>>> O_CLOEXEC
>>>>>      description.
>>>>>
>>>>> https://www.python.org/dev/peps/pep-0446/
>>>>> _______________________________________________
>>>>> CentralOH mailing list
>>>>> CentralOH at python.org
>>>>> https://mail.python.org/mailman/listinfo/centraloh
>>>>>
>>>>
>>>>
>>>
>>> _______________________________________________
>>> CentralOH mailing list
>>> CentralOH at python.org
>>> https://mail.python.org/mailman/listinfo/centraloh
>>>
>>>
>>
>> _______________________________________________
>> CentralOH mailing list
>> CentralOH at python.org
>> https://mail.python.org/mailman/listinfo/centraloh
>>
>>
>
> _______________________________________________
> CentralOH mailing list
> CentralOH at python.org
> https://mail.python.org/mailman/listinfo/centraloh
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/centraloh/attachments/20180219/91343f3d/attachment-0001.html>

From joe at joeshaw.org  Mon Feb 19 17:54:37 2018
From: joe at joeshaw.org (Joe Shaw)
Date: Mon, 19 Feb 2018 22:54:37 +0000
Subject: [CentralOH] Popen file descriptor difference between Python 2.6
 and 2.7+ on Linux
In-Reply-To: <CAJzarMNvKKDVOJAHTgg41nO0UjbZxpuix0_vk3h5=kaQ9XNppw@mail.gmail.com>
References: <CAJzarMMSBicLqwiaULKupvYL5vHp7ptx6jMPjyaDEyJKEpsBiw@mail.gmail.com>
 <20180216233241.158f64c222273d749fe75ec7@columbus.rr.com>
 <CAJShLf9eqTWJUSci3WziPwAk1eF3yeqcw6xNRDTTGXY8hO=wqg@mail.gmail.com>
 <CAJShLf_BUAGigV_LDKE7hwnR0Fp5EhZ4ZZB=3Ee57AnwABPdzA@mail.gmail.com>
 <CAJzarMObD6nJPJ3fo13c+jEtagkySh0fXrm7YwhbX318dPqRQQ@mail.gmail.com>
 <CAJShLf-=DaTtMs+o+cannpGUwWUydnwskpyE1LcQi7UgkG5_AA@mail.gmail.com>
 <CAJzarMNvKKDVOJAHTgg41nO0UjbZxpuix0_vk3h5=kaQ9XNppw@mail.gmail.com>
Message-ID: <CAOxvq7xvN71jrM=JV0vCihMFVvC34Sx2L+k4ygebfbNuOaUMtw@mail.gmail.com>

Hi,

If you have access to the shell, bash can do a nice job of this
concurrently:

    diff -u <(gunzip -c file1.gz) <(gunzip -c file2.gz)

Under the covers it?s doing the /proc/fd trick, and it saves you from
having to use a temporary file or reimplementing it yourself in Python.

Joe

On Mon, Feb 19, 2018 at 3:55 PM Eric Floehr <eric at intellovations.com> wrote:

> Good ideas... the actual application is gunzipping two ASCII files
> (gzipp'ed for disk space consideration) and diff'ing them.
>
> These files are added to but not just appended to, so we need to find the
> changes between two files. We do that, rather than processing the whole
> file each time because these are large files with lots of row and parsing
> and determining if a row's data exists or has changed would be hugely
> expensive. And the system diff program is the best for that.
>
>
>
>
>
> On Sat, Feb 17, 2018 at 6:42 PM, Thomas Winningham <winningham at gmail.com>
> wrote:
>
>> Well, sucks you had to go through this rewrite. I think it shows that all
>> code has a certain level of implicit functionality whose interpretation
>> varies between stakeholders over time. So while it took a little digging, I
>> think it also shows that everything is eventually knowable if needed.
>>
>> "Explicit is better than implicit" although the original code seemed
>> rather explicit its own right, this new code is certainly more explicit. If
>> IO performance becomes an issue with the tempfile, maybe named pipes could
>> be used? I learned a lot about per-process file descriptors though, and
>> reinforces how recently I learned that *nix opens a file based on
>> descriptor and not by name whereas like you say Windows can only have a
>> file opened by one process at a time because the file name is the
>> descriptor in a sense.
>>
>> There's always a way to further over engineer this probably too, I'm
>> thinking with sockets or even a message queue :P
>>
>>
>> On Sat, Feb 17, 2018 at 11:20 AM, Eric Floehr <eric at intellovations.com>
>> wrote:
>>
>>> Thank you Neil and Thomas, here is a summary of what you discovered:
>>>
>>> There is a regression in Python 2.7 where file descriptors are not
>>> passed to child processes in the subprocess module, whereas in Python 2.6
>>> they were.
>>>
>>> In order to address this, they added a "pass_fds" parameter to Popen in
>>> Python 3.2. However Python 2.7 does not have this parameter.
>>>
>>> So, in order for this to work on Python 2.7, I resorted to using named
>>> temporary files. This code won't work on Windows NT and higher (per Python
>>> documentation, because named temporary files can't be opened more than once
>>> on Windows NT and higher but can on UNIX-like systems).
>>>
>>> Here is the summary of changes:
>>>
>>> 1. Wrote contents of first Popen's stdout to a NamedTemporaryFile
>>> 2. Passed the name of the NamedTemporaryFile to the second Popen
>>> 3. Added with blocks to ensure both Popen's get closed properly
>>> 4. Closed the NamedTemporaryFile after the second Popen finished, to
>>> delete the temp file off disk.
>>>
>>> Here is the code:
>>>
>>> --------------------------------------------------
>>> from subprocess import Popen, PIPE
>>> import tempfile
>>>
>>> with Popen(
>>>         ["cat"],
>>>         stdin=open('/tmp/test', 'rb'),
>>>         stdout=PIPE,
>>>         universal_newlines=True).stdout as data:
>>>     tmpfile = tempfile.NamedTemporaryFile()
>>>     tmpfile.write(data.read())
>>>     tmpfile.flush()
>>>
>>> with Popen(
>>>         ["cat",
>>>         tmpfile.name],
>>>         stdout=PIPE,
>>>         universal_newlines=True,).stdout as fddata:
>>>     print(fddata.read())
>>>
>>> tmpfile.close()
>>> --------------------------------------------------
>>>
>>>
>>> This works on 2.6, 2.7, and 3.5. Thanks again for all of your help!
>>> Eric
>>>
>>>
>>>
>>> On Sat, Feb 17, 2018 at 12:39 AM, Thomas Winningham <
>>> winningham at gmail.com> wrote:
>>>
>>>> oh and the shell=False and stderr = PIPE are not needed that was me
>>>> just trying to fudge things somehow and left those in there in that code...
>>>> really the pass_fds flag is the thing
>>>>
>>>> On Sat, Feb 17, 2018 at 12:38 AM, Thomas Winningham <
>>>> winningham at gmail.com> wrote:
>>>>
>>>>> I was replying on Twitter, but I suspect the behavior slowly became
>>>>> undefined somewhere between 2.7 and 3 ... but after 3.2 i can do this and
>>>>> it works:
>>>>> ------------------------
>>>>> from subprocess import Popen, PIPE
>>>>>
>>>>> data = Popen(
>>>>>     ["cat"],
>>>>>     stdin=open('/tmp/test', 'rb'),
>>>>>     stdout=PIPE,
>>>>>     stderr=PIPE,
>>>>>     universal_newlines=True,
>>>>>     shell=False).stdout
>>>>>
>>>>> fd_name = '/dev/fd/%d' % data.fileno()
>>>>>
>>>>> fddata = Popen(
>>>>>     ["cat",fd_name],
>>>>>     stdout=PIPE,
>>>>>     universal_newlines=True,
>>>>>     pass_fds=(data.fileno(),)).stdout
>>>>>
>>>>> print(fddata.read())
>>>>> ------------------------
>>>>> the "pass_fds" option was added in 3.2 per the subprocess
>>>>> documentation
>>>>>
>>>>>
>>>>> On Fri, Feb 16, 2018 at 11:32 PM, Neil Ludban <nludban at columbus.rr.com
>>>>> > wrote:
>>>>>
>>>>>> On Fri, 16 Feb 2018 22:34:35 -0500
>>>>>> Eric Floehr <eric at intellovations.com> wrote:
>>>>>> > I have a really odd problem. I have some code that requires a Linux
>>>>>> file
>>>>>> > descriptor (/dev/fd/N) where N is some number. The following
>>>>>> example code
>>>>>> > demonstrates effectively what is being done.
>>>>>> >
>>>>>> > In order to run the code create a file called "/tmp/test" with some
>>>>>> text in
>>>>>> > it.
>>>>>> >
>>>>>> > The code works in Python 2.6, but in Python 2.7 or later, I get:
>>>>>> >
>>>>>> > cat: /dev/fd/4: No such file or directory
>>>>>>
>>>>>> Smells like a back-porting of adding the O_CLOEXEC flag...
>>>>>>
>>>>>>      Unless O_CLOEXEC flag was specified, the new descriptor is set
>>>>>> to remain
>>>>>>      open across execve(2) system calls; see close(2), fcntl(2) and
>>>>>> O_CLOEXEC
>>>>>>      description.
>>>>>>
>>>>>> https://www.python.org/dev/peps/pep-0446/
>>>>>> _______________________________________________
>>>>>> CentralOH mailing list
>>>>>> CentralOH at python.org
>>>>>> https://mail.python.org/mailman/listinfo/centraloh
>>>>>>
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> CentralOH mailing list
>>>> CentralOH at python.org
>>>> https://mail.python.org/mailman/listinfo/centraloh
>>>>
>>>>
>>>
>>> _______________________________________________
>>> CentralOH mailing list
>>> CentralOH at python.org
>>> https://mail.python.org/mailman/listinfo/centraloh
>>>
>>>
>>
>> _______________________________________________
>> CentralOH mailing list
>> CentralOH at python.org
>> https://mail.python.org/mailman/listinfo/centraloh
>>
>>
> _______________________________________________
> CentralOH mailing list
> CentralOH at python.org
> https://mail.python.org/mailman/listinfo/centraloh
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/centraloh/attachments/20180219/2eb5d2f6/attachment-0001.html>