Large image pyramids using dask

[Don Venable]

Thanks for the feedback, Matt. Will check this out later this week. 



On Friday, August 14, 2015 at 8:29:17 PM UTC-5, Matthew Rocklin wrote:
>
> Nice work.  A couple of random notes from a dask.array perspective.
>
> The pyramid_reduce_hdf5 function loads the data from disk, then writes it 
> back out.  We call this many times with successively smaller datasets.  It 
> might be nice to read the data only once during this process and do all of 
> the writing in a single stream.  To do this you might want to look at the 
> `da.to_hdf5` function rather than the `.to_hdf5` method.
>
> Note that the map_overlap method overlaps a bit of every block from each 
> of its neighbors.  This effectively induces a copy which may be slow-ish in 
> some cases.  If your resizing is aligned very nicely with the block 
> structure then this overlap may not be necessary.  You might find the 
> `da.coarsen` function of interest in this case.
>
> Looking forward to seeing what happens with this,
> -matt
>
>
> On Friday, August 14, 2015 at 12:59:06 PM UTC-7, Don Venable wrote:
>>
>> I've been using scikit-image and dask to play with very large image 
>> rasters, and I have a gist for creating an image-pyramid using dask. As the 
>> chunk-shapes change during downsampling, you I couldn't use the 
>> apply_parallel() function from the existing PR on using dask + 
>> scikit-image. Was talking with @blink1073 and he mentioned that it might be 
>> useful/interesting to the community, so wanted to see what you thought. 
>>
>> Here's a link: 
>>
>> https://gist.github.com/venabled/669caeb2615e9ed1d952
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scikit-image/attachments/20150817/ea529b2b/attachment.html>