[Image-SIG] Workarounds for MemoryErrors

[Robert]

If you have a 64 bit OS, you can try Python mmap / numpy.memmap 
with Python2.6. And mmap objects have also the new buffer 
interface. Didn't try it so far with that sizes.

PIL says in Image.frombuffer's doc that it can directly reuse 
existing buffers:
"For some modes, the image memory will share memory with the 
original buffer (this means that changes to the original buffer 
object are reflected in the image). Not all modes can share 
memory; supported modes include "L", "RGBX", "RGBA", and "CMYK"."
However it didn't work when I tried it recently - both with normal 
buffers (array.array / numpy.ndarray) and mmap. But buffer(a) 
tells about 'read-only'. Maybe with Py2.7 / Py3 ..

Yet with numpy / scipy.ndimage you can perhaps also do your 
"collaging" neatly.

Otherwise with 32bit (wonder who is using such big image), and in 
case its just about sticking rectangles and simple resizing, you 
can perhaps simply do some iteration over lines on 
file.seek(x+y*cx...) level


Robert


B. Bogart wrote:
> Thanks all for your help.
> 
> The code generates a large image from many small ones.
> 
> So I'll just scale each image individually before collaging them.
> 
> .b.
> 
> Gregor Kopka wrote:
>> Hello,
>>
>> to process a ~ 25GB (uncompressed image data) you are afaik out of luck
>> with PIL.
>> Only chance would be to use a fairly recent 64 bit OS with 64 bit python
>> binaries (some custom patches /might/ be needed though) and give it a
>> hefty amount of swapspace (at least eight times main memory) to fit the
>> image into virtual memory, but then you would wait for the disk for
>> hours (at best) or - depending on the operations you want to perform -
>> way longer.
>>
>> The question of how to open the resulting image also arises...
>>
>> Since you didn't give any details on what you need as output format, or
>> the operations you want to perform on the image while in memory (or
>> whereever it might be at that point) it's hard to give any better clues.
>>
>> Regards,
>>
>> Gregor
>>
>> B. Bogart schrieb:
>>> Hello all,
>>>
>>> I want to create a very large RGBA image (96000x72000 pixels).
>>>
>>> I have 4GB of RAM.
>>>
>>> Is there an easy way of getting around this error by having PIL only
>>> allocate one section of the image at a time?
>>>
>>> If PIL does not have any internal trick to work with large images then
>>> I'll have to make 4+ smaller images one at a time, but then I'm not sure
>>> how I could combine them without needing to allocate a memory chunk for
>>> the whole image.
>>>
>>> Otherwise I suppose I'll have to try with some other language, perhaps
>>> C/SDL, though a quick calculation seems to show that such a large RGBA
>>> image is just unworkable. Is there some way of using disk space rather
>>> than memory? Does not matter if it is slow, just that it is possible!
>>>
>>> Any advice?
>>>
>>> Thanks,
>>> B.
>>> _______________________________________________
>>> Image-SIG maillist  -  Image-SIG at python.org
>>> http://mail.python.org/mailman/listinfo/image-sig
>>>
>>>   
> _______________________________________________
> Image-SIG maillist  -  Image-SIG at python.org
> http://mail.python.org/mailman/listinfo/image-sig
>