use of subprocess module inside generator

[Gary Herron]

On 05/13/2015 12:43 PM, Peter wrote:
>
> I'm using Python 3.4.3 on Windows 7 (with latest patches) to develop a 
> sqlcmd module for accessing SQL Server (via Microsoft's sqlcmd.exe).  
> My goal is to develop a 100% Python 3 module that's easy to use, 
> flexible, and by design shifts the majority of future SQL Server 
> Python database access maintenance to Microsoft.  So far, I've 
> successfully and quickly converted a few of my Python pyodbc 
> applications/tools to use this new module.  However, I recently ran 
> into an apparent Python issue which cost me a few hours to diagnose 
> and work around.

I doubt that you've hit a bug -- we've all done this kind of thing many 
times, and there's certainly no restriction on making procedure calls 
within a generator -- so that's probably not the problem either.

My guess is that you've misinterpreted the failure of the original 
code.  I don't know how that might be, but I do spot one oddity in your 
original code which may be responsible.  See below ...

>
> I'm hoping that someone might know what the root cause of my issue 
> was.  Perhaps I've hit a bug/restriction with Python generators?
>
> My original generator function looked like this:
>
> def _raw_data(cl, stdout, *, opath=None, timeout=timeout):
>     stdout = subprocess.check_output(cl, universal_newlines=True, 
> timeout=timeout)

This seems muddled -- you pass in a parameter, stdout, only to 
immediately overwrite its value with the output of check_output. What 
was in stdout originally, and more importantly, did you expect the newly 
assigned value from check_output to be returned to the calling 
procedure?  If so, that's your bug, because parameters in function calls 
don't work that way.

This makes sense with your workaround, since the assignment to stdout is 
preserved when done outside the function.

I hope that helps.

Gary Herron



>
>     if opath is None:
>         for line in stdout.splitlines():
>             yield line.strip()
>     else:
>         with open(opath) as f:
>             for line in f:
>                 yield line.strip()
>
> The above function appeared to work fine, if the command line directed 
> sqlcmd.exe to send its output to stdout.  However, if the command line 
> directed sqlcmd.exe to send its output to a file, 
> subprocess.check_output would never be called when next was called on 
> the returned generator.  I verified this behavior with print 
> statements inside my code, as well as, inside the subprocess module.
>
> My work around was to simply move the call to subprocess.check_output 
> outside of the generator function (see below) to its caller (a 
> non-generator function).  With this minor change, everything appears 
> to work as expected.  OK, so am I missing something here?
>
> def _raw_data(stdout, *, opath=None):
>     if opath is None:
>         for line in stdout.splitlines():
>             yield line.strip()
>     else:
>         with open(opath) as f:
>             for line in f:
>                 yield line.strip()
>
> Thank you in advance for your assistance.
>
> Peter Santoro


-- 
Dr. Gary Herron
Department of Computer Science
DigiPen Institute of Technology
(425) 895-4418