[Neuroimaging] Thread-safe ArrayProxy/fileslice?
paul mccarthy
pauldmccarthy at gmail.com
Wed Feb 8 16:12:25 EST 2017
Hi Nathaniel,
True - perhaps such a method could be added to the Opener class (with the
internal seek/read calls protected by a mutex), and used by read_segments,
instead of separate seek/read calls. This would be a very simple change -
I'm happy to prepare a PR if it sounds like a good idea.
Cheers,
Paul
On 8 February 2017 at 20:23, Nathaniel Smith <njs at pobox.com> wrote:
> The best solution would be to use a pread-style interface that lets you
> read from a given offset without seeking at all, and thus without needing
> any locks. Unfortunately, Python hasn't historically had the best support
> for this, but it's available if you're using Python 3 on unix:
> https://docs.python.org/3/library/os.html#os.pread
>
> -n
>
> On Feb 8, 2017 5:55 AM, "paul mccarthy" <pauldmccarthy at gmail.com> wrote:
>
>> Howdy all,
>>
>> Does anybody have experience accessing image data through the ArrayProxy
>> class (or functions in the fileslice module) in a multi-threaded
>> environment? I am visualising large 4D images which are kept on disk (via
>> my indexed_gzip module), and am having trouble when accessing the data
>> from multiple threads, as the seek/read pairs from different threads
>> will occasionally become intertwined with each other.
>>
>>
>> My hacky workaround is to patch the ArrayProxy.__getitem__ method, and
>> add a threading.Lock to each instance, as follows:
>>
>>
>> import threading
>>
>> import nibabel as nib
>> import nibabel.arrayproxy as ap
>>
>> def ArrayProxy__getitem__(self, slc):
>>
>> if not hasattr(self, '_thread_lock'):
>> return self.__real_getitem__(slc)
>>
>> self._thread_lock.acquire()
>>
>> try:
>> return ap.ArrayProxy.__real_getitem__(self, slc)
>>
>> finally:
>> self._thread_lock.release()
>>
>> # Patch ArrayProxy.__getitem__
>> ap.ArrayProxy.__real_getitem__ = ap.ArrayProxy.__getitem__
>> ap.ArrayProxy.__getitem__ = ArrayProxy__getitem__
>>
>>
>> # Then add a lock to instances
>> # which need to be thread-safe
>> img = nib.load('MNI152_T1_2mm.nii.gz')
>> img.dataobj._thread_lock = threading.Lock()
>>
>>
>> This is the first thing I came up with, although I will probably end up
>> adding the lock to the fileobj, and patching the fileslice.fileslice
>> function instead. Unless there are any better ideas?
>>
>> Cheers,
>>
>> Paul
>>
>> _______________________________________________
>> Neuroimaging mailing list
>> Neuroimaging at python.org
>> https://mail.python.org/mailman/listinfo/neuroimaging
>>
>>
> _______________________________________________
> Neuroimaging mailing list
> Neuroimaging at python.org
> https://mail.python.org/mailman/listinfo/neuroimaging
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/neuroimaging/attachments/20170208/5a832553/attachment.html>
More information about the Neuroimaging
mailing list