[Neuroimaging] Thread-safe ArrayProxy/fileslice?

[Christopher Markiewicz]

That sounds reasonable to me.

On Wed, Feb 8, 2017 at 4:12 PM, paul mccarthy <pauldmccarthy at gmail.com>
wrote:

> Hi Nathaniel,
>
> True - perhaps such a method could be added to the Opener class (with the
> internal seek/read calls protected by a mutex), and used by read_segments,
> instead of separate seek/read calls. This would be a very simple change -
> I'm happy to prepare a PR if it sounds like a good idea.
>
> Cheers,
>
> Paul
>
> On 8 February 2017 at 20:23, Nathaniel Smith <njs at pobox.com> wrote:
>
>> The best solution would be to use a pread-style interface that lets you
>> read from a given offset without seeking at all, and thus without needing
>> any locks. Unfortunately, Python hasn't historically had the best support
>> for this, but it's available if you're using Python 3 on unix:
>> https://docs.python.org/3/library/os.html#os.pread
>>
>> -n
>>
>> On Feb 8, 2017 5:55 AM, "paul mccarthy" <pauldmccarthy at gmail.com> wrote:
>>
>>> Howdy all,
>>>
>>> Does anybody have experience accessing image data through the ArrayProxy
>>> class (or functions in the fileslice module) in a multi-threaded
>>> environment? I am visualising large 4D images which are kept on disk (via
>>> my indexed_gzip module), and am having trouble when accessing the data
>>> from multiple threads, as the seek/read pairs from different threads
>>> will occasionally become intertwined with each other.
>>>
>>>
>>> My hacky workaround is to patch the ArrayProxy.__getitem__ method, and
>>> add a threading.Lock to each instance, as follows:
>>>
>>>
>>> import threading
>>>
>>> import nibabel            as nib
>>> import nibabel.arrayproxy as ap
>>>
>>> def ArrayProxy__getitem__(self, slc):
>>>
>>>     if not hasattr(self, '_thread_lock'):
>>>         return self.__real_getitem__(slc)
>>>
>>>     self._thread_lock.acquire()
>>>
>>>     try:
>>>         return ap.ArrayProxy.__real_getitem__(self, slc)
>>>
>>>     finally:
>>>         self._thread_lock.release()
>>>
>>> # Patch ArrayProxy.__getitem__
>>> ap.ArrayProxy.__real_getitem__ = ap.ArrayProxy.__getitem__
>>> ap.ArrayProxy.__getitem__      = ArrayProxy__getitem__
>>>
>>>
>>> # Then add a lock to instances
>>> # which need to be thread-safe
>>> img = nib.load('MNI152_T1_2mm.nii.gz')
>>> img.dataobj._thread_lock = threading.Lock()
>>>
>>>
>>> This is the first thing I came up with, although I will probably end up
>>> adding the lock to the fileobj, and patching the fileslice.fileslice
>>> function instead. Unless there are any better ideas?
>>>
>>> Cheers,
>>>
>>> Paul
>>>
>>> _______________________________________________
>>> Neuroimaging mailing list
>>> Neuroimaging at python.org
>>> https://mail.python.org/mailman/listinfo/neuroimaging
>>>
>>>
>> _______________________________________________
>> Neuroimaging mailing list
>> Neuroimaging at python.org
>> https://mail.python.org/mailman/listinfo/neuroimaging
>>
>>
>
> _______________________________________________
> Neuroimaging mailing list
> Neuroimaging at python.org
> https://mail.python.org/mailman/listinfo/neuroimaging
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/neuroimaging/attachments/20170208/0893569e/attachment.html>