[Image-SIG] Corruption writing PNG data

[Ben Taylor]

Hi Eric (again CC list)

Thanks very much for looking into this, I appreciate it! One of my
colleagues mentioned Pillow to me when I brought this up here originally
(we're currently running Fedora 17, I gather later versions use Pillow
instead of PIL so at some point soon we'll migrate I'd imagine, but
we'll need to change our codebase a bit).

I've tried your test code but the workaround doesn't work for me - no
setting of compress_level gives me a working PNG. Version 1.1.7 doesn't
support compress_level for PNG images that I can see (not in the
handbook for 1.1.6 nor in the PngImagePlugin source for 1.1.7), so I
suspect it's silently ignoring the compress_level anyway - I don't know
if you're running a tweaked version of PIL. I'm not au-fait enough with
PIL, zlib or PNG images to try and hack that in myself!

I'm out of the office for a few days after today, so I'll respond next
week if you find anything. Thanks again!

Cheers
Ben

On 20/08/13 06:16, Eric Soroos wrote:
> Hi Ben.
> 
> (mirrored to the list, who knows when it'll get delivered)
> 
> I'm a maintainer for Pillow, an actively maintained successor for PIL
> that has some bugfixes and new features.
> I've taken a quick look, and it appears that the bug is in Pillow as well.
> 
> It appears that at some point, once the file gets over about 23 mp, or
> at 8285x2780, there's a failure. It works at 8284x2780, and also if the
> subselected region is panned. I've traced it down to the calls to zlib
> -- Pillow/Pil is calling it to compress the png formatted scanlines, and
> somewhere in there (but not at the end of the image) the compressed
> datastream is getting corrupted.
> 
> However, there is a small workaround that appears to help. If a
> compress_level is passed into the save call, requesting anything from 0
> (no compression) to 9 (max), it saves properly. Passing in -1 (the
> default compression level, equivalent to 6) triggers the failure.
> 
> I've been testing it with the following:
> (if maxblock is big enough, you get all of the image data in one IDAT
> block, and we can test the decompression of it with the plain zlib
> decompression call. It's commented out here.)
> 
> 
> from PIL import Image, ImageFile
> import zlib
> 
> npz = numpy.load("data.npz")
> imagedata = npz['arr_0']
> palette = npz['arr_1']
> 
> Image.DEBUG = 1
> #ImageFile.MAXBLOCK = 512*1024
> 
> print(imagedata.shape)
> 
> def test(p):
>     i1 = imagedata[0:2780,0:p]
>     im = Image.fromarray(i1, 'P')
>     im.putpalette(palette)
>     print (im)
>     im.save('tmp.png',compress_level=9 )
>     im2 = Image.open('tmp.png')
>     print (im2)
>     print ("Verify: %s" %im2.verify())
>     try:
>         im2 = Image.open('tmp.png')
>         ImageFile.LOAD_TRUNCATED_IMAGES
>         im2.load()
>         print ("%s success" %p)
>     except:
>         print ("FAIL: %s" %p)
>         #raise
> 
>     #with open('tmp.png','rb') as f:
>     #    f.seek(821)#
> 
>     #    s = zlib.decompress(f.read(385128))
>     #    print ("successful decompress")
> 
> 
> test(8284)
> test(8285)
> 
> The error I'm getting: "zlib.error: Error -3 while decompressing data:
> invalid distances set" would indicate that the compressed data stream is
> either corrupt or was compressed with different settings, perhaps a
> larger compression window.
> 
> I'm hoping to find the actual bug here, rather than a vague workaround.
> 
> eric
> 
> 
> On 07/26/2013 04:55 AM, Ben Taylor wrote:
>> Hi all
>>
>> I've got some data that should be fine (came from a known-good netCDF
>> file) but causes PIL to write an invalid image if it is used to save in
>> PNG format (it's happy writing GIF format).
>>
>> The same code writes PNGs quite happily 99% of the time, but just
>> occasionally we generate a netCDF that causes this bug - we don't know
>> why. We're not sure if the problem is actually in PIL or if it might be
>> in libpng. Anyone mind taking a look please?
>>
>> Test data at ftp://rsg.pml.ac.uk/rsg/benj/pil_problem/:
>> data.npz - Numpy npz data with data that causes the issue
>> working_data.npz - npz file from another source that works fine
>> test_harness.py - Run this with the two test files to demonstrate the
>> problem - the PNG generated from data.npz is corrupt.
>>
>> Test harness code below (just to demonstrate it's nothing complicated).
>>
>> TIA
>> Ben
>>
>> #!/usr/bin/env python
>>
>> import numpy
>> import Image
>>
>> def test(infile, prefix):
>>
>>     npz = numpy.load(infile)
>>     imagedata = npz['arr_0']
>>     palette = npz['arr_1']
>>
>>     out_image = Image.fromarray(imagedata, 'P')
>>     out_image.putpalette(palette)
>>
>>     out_image.save(prefix + ".gif") # ok
>>     out_image.save(prefix + ".png") # bust
>> # end function
>>
>> test("data.npz", "broken")
>> test("working_data.npz", "working")
>>
> 

-- 
Ben Taylor <benj at pml.ac.uk>, http://rsg.pml.ac.uk/
Remote Sensing Group, Plymouth Marine Laboratory
Tel: +44 (0)1752 633432, Fax: +44 (0)1752 633101


Latest news: www.pml.ac.uk and @PlymouthMarine

Plymouth Marine Laboratory (PML) is a company limited by guarantee registered in England & Wales, company number 4178503. Registered Charity No. 1091222. Registered Office: Prospect Place, The Hoe, Plymouth  PL1 3DH, UK. 

This message is private and confidential. If you have received this message in error, please notify the sender and remove it from your system. You are reminded that e-mail communications are not secure and may contain viruses; PML accepts no liability for any loss or damage which may be caused by viruses.