[python-win32] optimizing code to read serial port

[Tony Cappellini]

Vernon

Your suggestion is interesting but not practical for our needs.
We are collecting data from many devices that have failed tests, and often
have 10 or more failed devices that we need to collect the data from. These
are scattered all over a large lab environment. I have a Raspberry Pi, and
may try your suggestion on 1 device just to see the result.

Do you have any ides why running a terminal program written in (presumably
C, mentioned in my original message) doesn't seem to suffer from the
problems that my python app does, even when transferring the data at much
higher baud rates?

>>You will soon need to be using another solution anyway.
We've been using USB-serial adapters for a long time.

Thanks


On Sat, Feb 15, 2014 at 2:32 AM, Vernon D. Cole <vernondcole at gmail.com>wrote:

> Tony:
>
>   Don't do it!
>
> Do not attempt to make your data transfer faster while using this
>  defective hardware design!  Trust me on this.  I have been doing serial
> port data transfer since 300 baud was fast.
>
> The problem with your data transfer speed is not your program
> optimization. It is contained in the sentence:  "The serial port on the
> device we are dumping data from has a 2-wire interface (no handshaking)."
> This means that the program servicing the serial port must be scheduled at
> time intervals less than the time needed to fill your FIFO buffer.  Using
> Windows, you cannot guarantee that with any degree of certainty --
> particularly not at higher data rates -- because you have no control over
> the hardware-level interface drivers.
>
> If you really want higher data rates, you must use a dedicated device to
> receive the data. May I humbly suggest that an hour of your programming
> time would purchase a single board computer with a Python engine? Look at
> http://www.raspberrypi.org/.  Let the SBC collect the data.  Transfer
> blocks from it to the Windows environment using a more reliable protocol.
>
> Consider that none of the new Windows boxes I saw on my last shopping trip
> had serial ports.  You will soon need to be using another solution anyway.
>
>
>
>
>
>
>
> On Sat, Feb 15, 2014 at 3:01 AM, Tony Cappellini <cappy2112 at gmail.com>wrote:
>
>>
>> <python-win32 at python.org>
>>
>> Regarding an in-house python program we use for dumping diagnostic data
>> over a serial port on Windows 7 ...
>>
>>
>> I'm trying to determine if it's possible optimize the python code so that
>> dumping the data will occur faster. Currently, it takes ~25 minutes at 9600
>> baud to dump 8-15 MB of data. Going to higher baud rates does reduce this
>> time, but depending on which PC is running the program, going to higher
>> baud rates often results in ill-formed data. The serial port on the device
>> we are dumping data from has a 2-wire interface (no handshaking). Sometimes
>> 115200 baud works fine, other times the data gets all jumbled. We've
>> decided to default to 9600 to avoid getting jumbled data.
>>
>> Compared to an openly available terminal program (Tera Term Pro) written
>> in a compiled language, our Python version is much slower, but easier to
>> program. Tera Term's scripting language isn't as easy to script with as
>> Python.
>>
>> At the lowest level of the Python app, the last call in python is
>> win32file.ReadFile( comport, bytes_, None ). I believe execution goes to
>> a PyWin32 extension module at this point.
>>
>> By default, 1 is passed in for the number of bytes to read. I thought
>> this was very inefficient,
>> so I've experimented with values like 2,4,8,16. I didn't want to go
>> higher than 16 because the serial port FIFO on many PCs isn't any larger
>> than 16.
>>
>> While using cProfile, I could see that the number of function calls
>> executed was cut in half,
>> each time the # of bytes read doubled. Normally this would be a win, but
>> the total time to dump the same amount of data over the serial port would
>> almost double also.
>>
>> While this may not be strictly a Python-specific issue, the performance
>> issue may be related to the Python-Win32 interface. For example, are there
>> more efficient ways to read data from a serial port using Python, than
>> calling win32file.ReadFile() ?
>>
>> I would also expect reading more 8-16 bytes at a time to be more
>> efficient than reading 1 byte.
>> I do expect that buffering weighs in heavily here, but I have no control
>> over what the serial hardware does nor how windows interacts with it.
>>
>> Having read several of the Python optimization tips, I've already
>> eliminated string concatenation, replaced globals variables with locals,
>> and eliminated several dot references that are inside of loops.There
>> doesn't seem to be any places I can use calls to map() or replace for loops
>> with list/gen comps.
>>
>> There is too much code in the app to post here. I know it's difficult to
>> comment on code that you can't see.
>>
>> The Python app is a single-threaded console app running on Python 2.7,
>> but we've seen the same performance issues as far back as Python 2.3 on
>> Windows XP.  I've finally decided to investigate this because users of the
>> program are complaining about the time to dump the data.
>>
>> I would appreciate your thoughts on this issue.
>>
>> thanks
>>
>> _______________________________________________
>> python-win32 mailing list
>> python-win32 at python.org
>> https://mail.python.org/mailman/listinfo/python-win32
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-win32/attachments/20140215/9b1adfed/attachment.html>