[Ironpython-users] Issues with multithreading, ironpython and numpy

[Keith Rome]

Your threading code isn't as correct or robust as you think it is, if you are getting service startup timeout errors. Windows assumes that the service control callbacks will be executed quickly without being blocked. Typically, the Start callback would be creating the service's main work thread, starting it (without waiting on anything from it), and then immediately returning to the SCM. This should generally be an activity that takes up to perhaps a second at most, but the SCM gives you a wide grace period - you should never be coming close to that timeout unless something really serious is wrong. Similarly, the Stop and Pause/Resume callbacks would typically use thread synchronization objects (Manual/Auto Reset Events are very common) to communicate the requested operation to the main work thread without blocking. Of course, this all means that your main work thread needs to be extremely robust and needs to regularly check for those event signals (and discontinue work with minimal latency) - which is not entirely trivial unless you have a fair amount [successful] of experience with it. SCM timeout errors  are indicating an insufficient service design.

Depending on what your scripts are doing have you considered just using Process.Start() to invoke them, or to invoke a small managed executable that wraps the script execution? There is more overhead with this approach but it is significantly less hassle (and more robust) than dealing with a service threading model if you aren't experienced in such things.


Keith Rome
Principal Architect @ Wintellect (www.wintellect.com<http://www.wintellect.com>)
770.617.4016 | krome at wintellect.com<mailto:rome at wintellect.com>

[cid:DB4B3CED-8E4A-4404-A5B0-2853E1AA014D]
Register today<https://www.wintellectnow.com/Account/Promo/ROME-13> for access to our high-quality on-demand training resources!
Use promotion code ROME-13 to activate your 14-day free trial.

From: Eirik Årdal <eirik.ardal at bouvet.no<mailto:eirik.ardal at bouvet.no>>
Date: Friday, October 18, 2013 4:20 AM
To: "ironpython-users at python.org<mailto:ironpython-users at python.org>" <ironpython-users at python.org<mailto:ironpython-users at python.org>>
Subject: [Ironpython-users] Issues with multithreading, ironpython and numpy

Hi

First, let me start by saying that I don't have a lot of experience with Iron Python, or Python for that matter, but I know a person involved with the current project who is an experienced Python developer, but is unable to help with this issue since we believe it has to do with our integration between my C# code and his Python code. Also, I am aware that this might not be directly a IronPython issue, but it's certainly involving IronPython, NymPy for .net and C# and integration between them.

The short version of our problem is that when I call his Python scripts through IronPython in a multithreaded context, I get a System.InsufficientMemoryException thrown by NumPyDotNet. However, if I run my code synchronously on one thread, it works fine.

The long version is that I have a Windows Service that constantly checks in a database for new entries that will trigger work to be done (calculating data). I need to be able to do the calculation work on another thread in order to not lock up the service, making it unresponsive. But calling the Python scripts through IronPython on my worker thread results in the exception above. If I run my code on one thread, the service is unresponsive during start-up and windows terminates it after a certain time, saying it could not put it into its "started" state since its locked by work.

I have tried various ways to do my work asynchronous, like starting a Task and using the timers in Microsoft's libraries. I have not tried async/await from .net 4.5 since upgrading to .net 4.5 is not desireable by the project management.

The same problem was explained (not by me) in this stackoverflow post http://stackoverflow.com/questions/10003344/ironpython-version-of-numpy-doesnt-work-in-multi-threaded-environment but there are very few other similar issues when I search forums.

Any suggestions or tips will be greatly appreciated.

Eirik Årdal / 54°24'42" S 3°25'33" E
Konsulent
Bouvet Norge AS, Solheimsgaten 15, NO-5058 Bergen
Mobil +47 992 999 31
http://www.bouvet.no<http://www.bouvet.no/?utm_source=Signatur%2Bno&utm_medium=Epost&utm_campaign=Bouvet%2Bsignatur>   eirik.ardal at bouvet.no<mailto:eirik.ardal at bouvet.no>

[bouvetlogomini]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20131018/9737c5ef/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 529A2558-817C-467F-8E7E-0789B90D675C[26].png
Type: image/png
Size: 6285 bytes
Desc: 529A2558-817C-467F-8E7E-0789B90D675C[26].png
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20131018/9737c5ef/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.jpg
Type: image/jpeg
Size: 1671 bytes
Desc: image003.jpg
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20131018/9737c5ef/attachment.jpg>