From winecoding at gmail.com Tue Dec 6 23:41:02 2016 From: winecoding at gmail.com (wine lover) Date: Tue, 6 Dec 2016 22:41:02 -0600 Subject: [scikit-image] how to correctly save an numpy array with float32 type into an image Message-ID: Dear All, In a program, I generate an numpy array, with shape (128,128), which is supposed to represent an image. For instance, I have an array temp_mask, which is of type float32 and shape (128,128), the maximum value is 1.0 and the minimum value is 0.0. I saved it using io.imsave(?mask_image?,temp_mask) However, after I re-opened this image using img_mask = io.imread(?mask_image?). The read image turns out to have type unit16, the max value becomes 65535 and the min value is 0 . It seems to me that io.imsave automatically transform the float32 array into an unit16 array. Is it possible to save the image while keeping the original type? If not, what?s the correct way to save an image represented as an array, with type float32 and the range of value [0.0,1.0]? Thank you very much! -------------- next part -------------- An HTML attachment was scrubbed... URL: From imanol.luengo at nottingham.ac.uk Wed Dec 7 05:54:16 2016 From: imanol.luengo at nottingham.ac.uk (Imanol Luengo) Date: Wed, 7 Dec 2016 10:54:16 +0000 Subject: [scikit-image] how to correctly save an numpy array with float32 type into an image In-Reply-To: References: Message-ID: <3c81b03b-4a0b-e792-a6e1-c97e70fe984a@nottingham.ac.uk> Hello, I would say there are two differences between 'Saving the data' and 'Displaying the data'. An image is discretized to `uint8` or `uint16` prior to being saved as standard formates (`.png` or `.jpg`). You could do something like ``` import numpy as np from skimage import io, util A = np.random.rand(100,100) io.imsave('tmp.png', A) B = util.img_as_float(io.imread('tmp.png') assert np.allclose(A, B) # ERROR ``` But you will find some discretization errors, which makes `B != A`. Having said that, if you want to preserve the data in `B`, I think the best option is to export the data using another format, e.g. numpy arrays: ``` import numpy as np A = np.random.rand(100,100) np.save('tmp.npy', A) B = np.load('tmp.npy') assert np.allclose(A, B) # True ``` Or alternatively, if you really want to save the data in a visualizable format, exporting the image as `.tif` format, which preserves data information, should also work: ``` A = np.random.rand(100,100) io.imsave('tmp.tif', A) B = io.imread('tmp.tif') assert np.allclose(A, B) # True ``` However, I would personally store my data in non-visualizable formats such as `.npy, .h5` (the later if you work with tons of data) as they usually offer another advantages (e.g. Datasets in HDF5). Hope it helps, Imanol On 07/12/16 04:41, wine lover wrote: > > Dear All, > > > In a program, I generate an numpy array, with shape (128,128), which > is supposed to represent an image. > > For instance, I havean array |temp_mask|, which is of type|float32 and > shape (128,128)|, the maximum value is|1.0|and the minimum value > is|0.0. |I saved it using|io.imsave(?mask_image?,temp_mask)|However, > after I re-opened this image using|img_mask = > io.imread(?mask_image?)|. The read image turns out to have > type|unit16|, the max value becomes|6553|5 and the min value is|0| . > It seems to me that io.imsave automatically transform the float32 > array into an unit16 array. > > Is it possible to save the image while keeping the original type? If > not, what?s the correct way to save an image represented as an array, > with type|float32|and the range of value |[0.0,1.0]|? > > > Thank you very much! > > > > _______________________________________________ > scikit-image mailing list > scikit-image at python.org > https://mail.python.org/mailman/listinfo/scikit-image This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please send it back to me, and immediately delete it. Please do not use, copy or disclose the information contained in this message or in any attachment. Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham. This message has been checked for viruses but the contents of an attachment may still contain software viruses which could damage your computer system, you are advised to perform your own checks. Email communications with the University of Nottingham may be monitored as permitted by UK legislation. -------------- next part -------------- An HTML attachment was scrubbed... URL: From winecoding at gmail.com Wed Dec 7 18:50:40 2016 From: winecoding at gmail.com (wine lover) Date: Wed, 7 Dec 2016 17:50:40 -0600 Subject: [scikit-image] how to correctly save an numpy array with float32 type into an image In-Reply-To: <3c81b03b-4a0b-e792-a6e1-c97e70fe984a@nottingham.ac.uk> References: <3c81b03b-4a0b-e792-a6e1-c97e70fe984a@nottingham.ac.uk> Message-ID: Hi Juan, Hi Imanol, Thank you so much for your replies, which are very helpful. During saving the image, I got some warning messages such as follows. What does it indicate? /development/lib/python3.4/site-packages/scikit_image-0.12.3-py3.4-linux-x86_64.egg/skimage/io/_io.py:132: UserWarning: /data/train/image_sampled.tif is a low contrast image warn('%s is a low contrast image' % fname) Thanks, Yuanyuan On Wed, Dec 7, 2016 at 4:54 AM, Imanol Luengo < imanol.luengo at nottingham.ac.uk> wrote: > Hello, > > I would say there are two differences between 'Saving the data' and > 'Displaying the data'. An image is discretized to `uint8` or `uint16` prior > to being saved as standard formates (`.png` or `.jpg`). You could do > something like > ``` > import numpy as np > from skimage import io, util > > A = np.random.rand(100,100) > io.imsave('tmp.png', A) > B = util.img_as_float(io.imread('tmp.png') > > assert np.allclose(A, B) # ERROR > ``` > > But you will find some discretization errors, which makes `B != A`. Having > said that, if you want to preserve the data in `B`, I think the best option > is to export the data using another format, e.g. numpy arrays: > ``` > import numpy as np > > A = np.random.rand(100,100) > np.save('tmp.npy', A) > B = np.load('tmp.npy') > > assert np.allclose(A, B) # True > ``` > > Or alternatively, if you really want to save the data in a visualizable > format, exporting the image as `.tif` format, which preserves data > information, should also work: > ``` > A = np.random.rand(100,100) > io.imsave('tmp.tif', A) > B = io.imread('tmp.tif') > > assert np.allclose(A, B) # True > ``` > > However, I would personally store my data in non-visualizable formats such > as `.npy, .h5` (the later if you work with tons of data) as they usually > offer another advantages (e.g. Datasets in HDF5). > > Hope it helps, > > Imanol > > On 07/12/16 04:41, wine lover wrote: > > Dear All, > > > In a program, I generate an numpy array, with shape (128,128), which is > supposed to represent an image. > > For instance, I have an array temp_mask, which is of type float32 and > shape (128,128), the maximum value is 1.0 and the minimum value is 0.0. I > saved it using io.imsave(?mask_image?,temp_mask) However, after I > re-opened this image using img_mask = io.imread(?mask_image?). The read > image turns out to have type unit16, the max value becomes 65535 and the > min value is 0 . It seems to me that io.imsave automatically transform > the float32 array into an unit16 array. > > Is it possible to save the image while keeping the original type? If not, > what?s the correct way to save an image represented as an array, with type > float32 and the range of value [0.0,1.0]? > > > Thank you very much! > > > _______________________________________________ > scikit-image mailing listscikit-image at python.orghttps://mail.python.org/mailman/listinfo/scikit-image > > > > > This message and any attachment are intended solely for the addressee > and may contain confidential information. If you have received this > message in error, please send it back to me, and immediately delete it. > > Please do not use, copy or disclose the information contained in this > message or in any attachment. Any views or opinions expressed by the > author of this email do not necessarily reflect the views of the > University of Nottingham. > > This message has been checked for viruses but the contents of an > attachment may still contain software viruses which could damage your > computer system, you are advised to perform your own checks. Email > communications with the University of Nottingham may be monitored as > permitted by UK legislation. > > > _______________________________________________ > scikit-image mailing list > scikit-image at python.org > https://mail.python.org/mailman/listinfo/scikit-image > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jni.soma at gmail.com Wed Dec 7 19:32:16 2016 From: jni.soma at gmail.com (Juan Nunez-Iglesias) Date: Thu, 8 Dec 2016 11:32:16 +1100 Subject: [scikit-image] how to correctly save an numpy array with float32 type into an image In-Reply-To: References: <3c81b03b-4a0b-e792-a6e1-c97e70fe984a@nottingham.ac.uk> Message-ID: <3977dbb8-b3e6-4956-b1ff-ba39e18742d2@Spark> Hi Yuanyuan, This warning is raised when the value range of a float image is less than 1/256, because in those cases, most image viewers will show the image as having a single shade of grey. For example, if you have an image of physical measurements in the range [0, 1e-5], and you save that as float, in many viewers it will appear as all black. It is not something to worry about if you just want to save the data for later. We implemented it to help you figure out what is happening when the image doesn?t ?look right? in a viewer. =) Juan. On 8 Dec. 2016, 10:51 AM +1100, wine lover , wrote: > Hi Juan, Hi Imanol, > > Thank you so much for your replies, which are very helpful. > > During saving the image, I got some warning messages such as follows. What does it indicate? > /development/lib/python3.4/site-packages/scikit_image-0.12.3-py3.4-linux-x86_64.egg/skimage/io/_io.py:132: UserWarning: /data/train/image_sampled.tif is a low contrast image > ? warn('%s is a low contrast image' % fname) > > Thanks, > Yuanyuan > > > On Wed, Dec 7, 2016 at 4:54 AM, Imanol Luengo wrote: > > > Hello, > > > I would say there are two differences between 'Saving the data' and 'Displaying the data'. An image is discretized to `uint8` or `uint16` prior to being saved as standard formates (`.png` or `.jpg`). You could do something like > > > ``` > > > import numpy as np > > > from skimage import io, util > > > > > > A = np.random.rand(100,100) > > > io.imsave('tmp.png', A) > > > B = util.img_as_float(io.imread('tmp.png') > > > > > > assert np.allclose(A, B) # ERROR > > > ``` > > > But you will find some discretization errors, which makes `B != A`. Having said that, if you want to preserve the data in `B`, I think the best option is to export the data using another format, e.g. numpy arrays: > > > ``` > > > import numpy as np > > > > > > A = np.random.rand(100,100) > > > np.save('tmp.npy', A) > > > B = np.load('tmp.npy') > > > > > > assert np.allclose(A, B) # True > > > ``` > > > Or alternatively, if you really want to save the data in a visualizable format, exporting the image as `.tif` format, which preserves data information, should also work: > > > ``` > > > A = np.random.rand(100,100) > > > io.imsave('tmp.tif', A) > > > B = io.imread('tmp.tif') > > > > > > assert np.allclose(A, B) # True > > > ``` > > > However, I would personally store my data in non-visualizable formats such as `.npy, .h5` (the later if you work with tons of data) as they usually offer another advantages (e.g. Datasets in HDF5). > > > Hope it helps, > > > Imanol > > > > > > On 07/12/16 04:41, wine lover wrote: > > > > Dear All, > > > > > > > > In a program, I generate an numpy array, with shape (128,128), which is supposed to represent an image. > > > > For instance, I have?an array temp_mask, which is of type?float32 and shape (128,128), the maximum value is?1.0?and the minimum value is?0.0. ?I saved it using?io.imsave(?mask_image?,temp_mask)?However, after I re-opened this image using?img_mask = io.imread(?mask_image?). The read image turns out to? have type?unit16, the max value becomes?65535 and the min value is?0?.? It seems to me that io.imsave automatically transform the float32 array into an unit16 array. > > > > Is it possible to save the image while keeping the original type? If not, what?s the correct way to save an image represented as an array, with type?float32 and the range of value [0.0,1.0]? > > > > > > > > Thank you very much! > > > > > > > > > > > > > > > > _______________________________________________ > > > > scikit-image mailing list > > > > scikit-image at python.org > > > > https://mail.python.org/mailman/listinfo/scikit-image > > > > > > > > > > > > > This message and any attachment are intended solely for the addressee > > > and may contain confidential information. If you have received this > > > message in error, please send it back to me, and immediately delete it. > > > > > > Please do not use, copy or disclose the information contained in this > > > message or in any attachment. Any views or opinions expressed by the > > > author of this email do not necessarily reflect the views of the > > > University of Nottingham. > > > > > > This message has been checked for viruses but the contents of an > > > attachment may still contain software viruses which could damage your > > > computer system, you are advised to perform your own checks. Email > > > communications with the University of Nottingham may be monitored as > > > permitted by UK legislation. > > > > > > _______________________________________________ > > > scikit-image mailing list > > > scikit-image at python.org > > > https://mail.python.org/mailman/listinfo/scikit-image > > > > > _______________________________________________ > scikit-image mailing list > scikit-image at python.org > https://mail.python.org/mailman/listinfo/scikit-image -------------- next part -------------- An HTML attachment was scrubbed... URL: From winecoding at gmail.com Sun Dec 11 01:46:04 2016 From: winecoding at gmail.com (wine lover) Date: Sun, 11 Dec 2016 00:46:04 -0600 Subject: [scikit-image] pixel value changes during reading image and showing images using skimage and matplotlib Message-ID: Dear All, I have a tif images, its type is float32, shape is (128*128) (a grayscale image). All the pixel values are of range [0.0, 1.0] I am trying to read it using skimage and show it on screen using matplotlib . from skimage import ioimport matplotlib.pyplot as plt output=io.imread(os.path.join(image_path,raw_image_name))print(output.dtype)print(output.shape)print(output.max())print(output.min()) plt.imshow(output) plt.show() The output image looks like color image instead of gray image as shown originally. I attached the screenshot as the capture-1.jpg. However, when I read the image using matplotlib instead,i.e., output=plt.imread(os.path.join(image_path,raw_image_name)). I found that pixel value will become 255 and 0. The dtype is still float32. But when I print output, the pixel values are either 0. or 255. The output image will become black as shown in the second image (capture-2.jpg). I am confused how does this work? My guess is that there are some dtype changes happening during the reading image and showing image, Thanks, Yuanyuan -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Capture-1.JPG Type: image/jpeg Size: 44467 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Capture_2.JPG Type: image/jpeg Size: 28963 bytes Desc: not available URL: From imanol.luengo at nottingham.ac.uk Sun Dec 11 05:54:49 2016 From: imanol.luengo at nottingham.ac.uk (Imanol Luengo) Date: Sun, 11 Dec 2016 10:54:49 +0000 Subject: [scikit-image] pixel value changes during reading image and showing images using skimage and matplotlib In-Reply-To: References: Message-ID: <9abc3f6b-497c-b2b2-bc43-7e39824af6cb@nottingham.ac.uk> Dear Yuanyuan, The reason why you are looking at a *color* image is due to the colormaps: http://matplotlib.org/users/colormaps.html The default colormap (to map from numbers to colors) is not grayscale, as matplotlib is not primarly focus on images, and grayscale is not an *adequate* colormap to represent other types of plots. To properly visualize your image specify a colormap to the plot function: plt.imshow(output, 'gray') Cheers, Imanol On 11/12/16 06:46, wine lover wrote: > Dear All, > > I have a tif images, its type is |float32|, shape is |(128*128)| (a > grayscale image). All the pixel values are of range |[0.0, 1.0]| > > I am trying to read it using |skimage| and show it on screen using > |matplotlib|. > > |fromskimage importio importmatplotlib.pyplot asplt > output=io.imread(os.path.join(image_path,raw_image_name))print(output.dtype)print(output.shape)print(output.max())print(output.min())plt.imshow(output)plt.show()| > > The output image looks like color image instead of gray image as shown > originally. I attached the screenshot as the capture-1.jpg. > > However, when I read the image using |matplotlib| instead,i.e., > |output=plt.imread(os.path.join(image_path,raw_image_name))|. I found > that pixel value will become 255 and 0. The dtype is still float32. > But when I print output, the pixel values are either 0. or 255. > > The output image will become black as shown in the second image > (capture-2.jpg). I am confused how does this work? My guess is that > there are some dtype changes happening during the reading image and > showing image, > > > Thanks, > > Yuanyuan > > > > _______________________________________________ > scikit-image mailing list > scikit-image at python.org > https://mail.python.org/mailman/listinfo/scikit-image This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please send it back to me, and immediately delete it. Please do not use, copy or disclose the information contained in this message or in any attachment. Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham. This message has been checked for viruses but the contents of an attachment may still contain software viruses which could damage your computer system, you are advised to perform your own checks. Email communications with the University of Nottingham may be monitored as permitted by UK legislation. -------------- next part -------------- An HTML attachment was scrubbed... URL: From winecoding at gmail.com Sun Dec 11 14:03:56 2016 From: winecoding at gmail.com (wine lover) Date: Sun, 11 Dec 2016 13:03:56 -0600 Subject: [scikit-image] pixel value changes during reading image and showing images using skimage and matplotlib In-Reply-To: <9abc3f6b-497c-b2b2-bc43-7e39824af6cb@nottingham.ac.uk> References: <9abc3f6b-497c-b2b2-bc43-7e39824af6cb@nottingham.ac.uk> Message-ID: Hi Imanol, It works fine now, thanks for your suggestions. Regarding the second part of my question. If I read the image using matlibplot instead of skimage, i.e., output=plt.imread(os.path.join(image_path,raw_image_name)) I found that the pixel values of output were all changed, and they becomes either 255. or 0. which causes the image shown as all black. What are the possible reasons for plt.imread. Thanks. Yuanyuan On Sun, Dec 11, 2016 at 4:54 AM, Imanol Luengo < imanol.luengo at nottingham.ac.uk> wrote: > Dear Yuanyuan, > > The reason why you are looking at a *color* image is due to the colormaps: > > http://matplotlib.org/users/colormaps.html > > The default colormap (to map from numbers to colors) is not grayscale, as > matplotlib is not primarly focus on images, and grayscale is not an > *adequate* colormap to represent other types of plots. > > To properly visualize your image specify a colormap to the plot function: > > plt.imshow(output, 'gray') > > Cheers, > > Imanol > > On 11/12/16 06:46, wine lover wrote: > > Dear All, > > I have a tif images, its type is float32, shape is (128*128) (a grayscale > image). All the pixel values are of range [0.0, 1.0] > > I am trying to read it using skimage and show it on screen using > matplotlib. > > from skimage import ioimport matplotlib.pyplot as plt > output=io.imread(os.path.join(image_path,raw_image_name))print(output.dtype)print(output.shape)print(output.max())print(output.min()) > plt.imshow(output) > plt.show() > > The output image looks like color image instead of gray image as shown > originally. I attached the screenshot as the capture-1.jpg. > > However, when I read the image using matplotlib instead,i.e., > output=plt.imread(os.path.join(image_path,raw_image_name)). I found that > pixel value will become 255 and 0. The dtype is still float32. But when I > print output, the pixel values are either 0. or 255. > > The output image will become black as shown in the second image > (capture-2.jpg). I am confused how does this work? My guess is that there > are some dtype changes happening during the reading image and showing > image, > > > Thanks, > > Yuanyuan > > > _______________________________________________ > scikit-image mailing listscikit-image at python.orghttps://mail.python.org/mailman/listinfo/scikit-image > > > > > This message and any attachment are intended solely for the addressee > and may contain confidential information. If you have received this > message in error, please send it back to me, and immediately delete it. > > Please do not use, copy or disclose the information contained in this > message or in any attachment. Any views or opinions expressed by the > author of this email do not necessarily reflect the views of the > University of Nottingham. > > This message has been checked for viruses but the contents of an > attachment may still contain software viruses which could damage your > computer system, you are advised to perform your own checks. Email > communications with the University of Nottingham may be monitored as > permitted by UK legislation. > > > _______________________________________________ > scikit-image mailing list > scikit-image at python.org > https://mail.python.org/mailman/listinfo/scikit-image > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From igor.karlovic at gmail.com Fri Dec 9 01:43:10 2016 From: igor.karlovic at gmail.com (Igor Karlovits) Date: Fri, 9 Dec 2016 07:43:10 +0100 Subject: [scikit-image] GLCM and angles Message-ID: Hello everybody I would have a question regarding the results of graycoprops which is giving me the same results for all attributes (energy, ASM etc...) in angles of rotation of 135 and 90. Angles 0, 45 and 90 are ok (with an offset of 1) but the angle 135 is giving the same results as 90? Any explanation of this? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Tue Dec 13 14:16:56 2016 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Tue, 13 Dec 2016 11:16:56 -0800 Subject: [scikit-image] scikit-image lecture at ImageXD, 29th of March Message-ID: <1481656616.408958.817892857.62660368@webmail.messagingengine.com> Hi, everyone We are looking for tutors from the scikit-image team to teach at the annual ImageXD conference (this year at the University of Washington), to be held on 29 March 2017. Please indicate your availability to me via private email. Thanks! St?fan From stefanv at berkeley.edu Sat Dec 17 17:16:04 2016 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Sat, 17 Dec 2016 14:16:04 -0800 Subject: [scikit-image] New team member Message-ID: <1590edb34a0.273e.acf34a9c767d7bb498a799333be0433e@fastmail.com> Hi everyone I would like to welcome the newest member of the scikit-image core team: Alexandre di Siqueira. We're excited to have you on board, Alex! Best regards St?fan From stefanv at berkeley.edu Sat Dec 17 17:24:18 2016 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Sat, 17 Dec 2016 14:24:18 -0800 Subject: [scikit-image] Release schedule Message-ID: <1590ee2be50.273e.acf34a9c767d7bb498a799333be0433e@fastmail.com> Hi, everyone It's time for a new skimage release. I have the following timeline in mind: Thursday 22 Dec, feature freeze -- bug fixes, documentation, clean up -- January 5th, tag pre-release -- developer and early adopter testing-- January 10th, tag release How does that sound? Also, do we have a volunteer for release manager? :) Finally, I m thinking it may be a good idea to make an automated release schedule for the future. E.g. release every 4 months, and work out how long we need to freeze each time. Opinions? St?fan From egor.v.panfilov at gmail.com Sat Dec 17 17:59:27 2016 From: egor.v.panfilov at gmail.com (Egor Panfilov) Date: Sun, 18 Dec 2016 01:59:27 +0300 Subject: [scikit-image] Release schedule Message-ID: Hi Stefan, >From my perspective there are still some issues to be closed before the release (see https://github.com/scikit-image/scikit-image/milestone/6). This year we have been using GitHub features extensively and they should represent our current point of view regarding the codebase status. Among the tagged issues there are some long-lasting ones, and these could be transfered into 0.13 roadmap. As for the others, I'd prefer to solve them first. Regarding release schedule: we are having a discussion in https://github.com/scikit-image/scikit-image/issues/2357. In short, the convenient release schedule should be tied to deprecation policy timeline, which is to be reviewed. Above said, I'd suggest to push the release date a bit further. At least, the freeze initiation. Regards, Egor Panfilov P.S. I'm traveling on the next week, and will not be able to contribute significant efforts till Dec'26th. However, I'll try to track the mailing list, and participate in the discussions (if any). 2016-12-18 1:24 GMT+03:00 Stefan van der Walt : > Hi, everyone > > It's time for a new skimage release. I have the following timeline in mind: > > Thursday 22 Dec, feature freeze > -- bug fixes, documentation, clean up -- > January 5th, tag pre-release > -- developer and early adopter testing-- > January 10th, tag release > > How does that sound? > > Also, do we have a volunteer for release manager? :) > > Finally, I m thinking it may be a good idea to make an automated release > schedule for the future. E.g. release every 4 months, and work out how long > we need to freeze each time. Opinions? > > St?fan > > > _______________________________________________ > scikit-image mailing list > scikit-image at python.org > https://mail.python.org/mailman/listinfo/scikit-image > -------------- next part -------------- An HTML attachment was scrubbed... URL: From siqueiraaf at gmail.com Tue Dec 20 05:16:48 2016 From: siqueiraaf at gmail.com (Alexandre Fioravante de Siqueira) Date: Tue, 20 Dec 2016 11:16:48 +0100 Subject: [scikit-image] New team member Message-ID: Dear St?fan and all, thank you very much! I'll do my best to help in the development, documentation and propagation of scikit-image! Thanks again, have a nice weekend! Alex -- ------------------------------------------------------------------------------ Dr. Alexandre 'Jaguar' Fioravante de Siqueira Office 306 - Institut f?r Geologie Fakult?t f?r Geowissenschaften, Geotechnik und Bergbau Technische Universit?t Bergakademie Freiberg Bernhard-von-Cotta-Stra?e, 2 Freiberg (09599), Mittelsachsen - Sachsen - Deutschland Lattes curriculum: http://lattes.cnpq.br/3936721630855880/ Personal site: http://www.programandociencia.com/about/ Github: http://www.github.com/alexandrejaguar/ Skype: alexandrejaguar Twitter: http://www.twitter.com/alexdesiqueira/ ------------------------------------------------------------------------------ -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Wed Dec 21 18:12:05 2016 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Wed, 21 Dec 2016 15:12:05 -0800 Subject: [scikit-image] Release schedule In-Reply-To: References: Message-ID: <1482361925.3134024.826460969.50EBCD6F@webmail.messagingengine.com> Hi Egor On Sat, Dec 17, 2016, at 14:59, Egor Panfilov wrote: > Above said, I'd suggest to push the release date a bit further. At > least, the freeze initiation. OK, how about the following: - Feature freeze 31 December 2016 - Tag pre-relerase 15 January 2017 - Tag release 20 January 2017 St?fan > Regards, > Egor Panfilov > > P.S. I'm traveling on the next week, and will not be able to > contribute significant efforts till Dec'26th. However, I'll try > to track the mailing list, and participate in the discussions (if > any). > > 2016-12-18 1:24 GMT+03:00 Stefan van der Walt : >> Hi, everyone >> >> It's time for a new skimage release. I have the following timeline >> in mind: >> >> Thursday 22 Dec, feature freeze >> -- bug fixes, documentation, clean up -- >> January 5th, tag pre-release >> -- developer and early adopter testing-- >> January 10th, tag release >> >> How does that sound? >> >> Also, do we have a volunteer for release manager? :) >> >> Finally, I m thinking it may be a good idea to make an automated >> release schedule for the future. E.g. release every 4 months, and >> work out how long we need to freeze each time. Opinions? >> >> St?fan >> >> >> _______________________________________________ >> scikit-image mailing list >> scikit-image at python.org >> https://mail.python.org/mailman/listinfo/scikit-image > _________________________________________________ > scikit-image mailing list > scikit-image at python.org > https://mail.python.org/mailman/listinfo/scikit-image -------------- next part -------------- An HTML attachment was scrubbed... URL: From jni.soma at gmail.com Wed Dec 21 18:59:52 2016 From: jni.soma at gmail.com (Juan Nunez-Iglesias) Date: Thu, 22 Dec 2016 10:59:52 +1100 Subject: [scikit-image] Release schedule In-Reply-To: <1482361925.3134024.826460969.50EBCD6F@webmail.messagingengine.com> References: <1482361925.3134024.826460969.50EBCD6F@webmail.messagingengine.com> Message-ID: <73c37ce4-b826-4026-a8c3-603a023b9e97@Spark> I like both timelines. =) On 22 Dec. 2016, 10:13 AM +1100, Stefan van der Walt , wrote: > Hi Egor > > On Sat, Dec 17, 2016, at 14:59, Egor Panfilov wrote: > > Above said, I'd suggest to push the release date a bit further. At least, the freeze initiation. > > OK, how about the following: > > - Feature freeze 31 December 2016 > - Tag pre-relerase 15 January 2017 > - Tag release 20 January 2017 > > St?fan > > > Regards, > > Egor Panfilov > > > > P.S. I'm traveling on the next week, and will not be able to contribute significant efforts till Dec'26th. However, I'll try to track the mailing list, and participate in the discussions (if any). > > > > 2016-12-18 1:24 GMT+03:00 Stefan van der Walt : > > > Hi, everyone > > > > > > It's time for a new skimage release. I have the following timeline in mind: > > > > > > Thursday 22 Dec, feature freeze > > > -- bug fixes, documentation, clean up -- > > > January 5th, tag pre-release > > > -- developer and early adopter testing-- > > > January 10th, tag release > > > > > > How does that sound? > > > > > > Also, do we have a volunteer for release manager? :) > > > > > > Finally, I m thinking it may be a good idea to make an automated release schedule for the future. E.g. release every 4 months, and work out how long we need to freeze each time. Opinions? > > > > > > St?fan > > > > > > > > > _______________________________________________ > > > scikit-image mailing list > > > scikit-image at python.org > > > https://mail.python.org/mailman/listinfo/scikit-image > > _______________________________________________ > > scikit-image mailing list > > scikit-image at python.org > > https://mail.python.org/mailman/listinfo/scikit-image > > _______________________________________________ > scikit-image mailing list > scikit-image at python.org > https://mail.python.org/mailman/listinfo/scikit-image -------------- next part -------------- An HTML attachment was scrubbed... URL: From egor.v.panfilov at gmail.com Thu Dec 22 03:08:03 2016 From: egor.v.panfilov at gmail.com (Egor Panfilov) Date: Thu, 22 Dec 2016 11:08:03 +0300 Subject: [scikit-image] Release schedule In-Reply-To: <1482361925.3134024.826460969.50EBCD6F@webmail.messagingengine.com> References: <1482361925.3134024.826460969.50EBCD6F@webmail.messagingengine.com> Message-ID: Hi Stefan, - Feature freeze 31 December 2016 > - Tag pre-relerase 15 January 2017 > - Tag release 20 January 2017 This one looks good to me. Are you interested in being release manager this time around? I'm not really familiar with the procedure, so maybe someone else could put a hat on the task of tagging, and deployment of the docs to the website. At the same time, I'd love to help with the reviews and the preparation of the release notes. To the core team: could someone, please, dedicate some time before the freeze date to review and merge as many PR from the 0.13 roadmap as possible (as I remember most of them are already finished in terms of code)? Thanks in advance! Regards, Egor Panfilov 2016-12-22 2:12 GMT+03:00 Stefan van der Walt : > Hi Egor > > On Sat, Dec 17, 2016, at 14:59, Egor Panfilov wrote: > > Above said, I'd suggest to push the release date a bit further. At least, > the freeze initiation. > > > OK, how about the following: > > - Feature freeze 31 December 2016 > - Tag pre-relerase 15 January 2017 > - Tag release 20 January 2017 > > St?fan > > Regards, > Egor Panfilov > > P.S. I'm traveling on the next week, and will not be able to contribute > significant efforts till Dec'26th. However, I'll try to track the mailing > list, and participate in the discussions (if any). > > 2016-12-18 1:24 GMT+03:00 Stefan van der Walt : > > Hi, everyone > > It's time for a new skimage release. I have the following timeline in mind: > > Thursday 22 Dec, feature freeze > -- bug fixes, documentation, clean up -- > January 5th, tag pre-release > -- developer and early adopter testing-- > January 10th, tag release > > How does that sound? > > Also, do we have a volunteer for release manager? :) > > Finally, I m thinking it may be a good idea to make an automated release > schedule for the future. E.g. release every 4 months, and work out how long > we need to freeze each time. Opinions? > > St?fan > > > _______________________________________________ > scikit-image mailing list > scikit-image at python.org > https://mail.python.org/mailman/listinfo/scikit-image > > *_______________________________________________* > scikit-image mailing list > scikit-image at python.org > https://mail.python.org/mailman/listinfo/scikit-image > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fboulogne at sciunto.org Mon Dec 26 09:07:34 2016 From: fboulogne at sciunto.org (=?UTF-8?Q?Fran=c3=a7ois_Boulogne?=) Date: Mon, 26 Dec 2016 15:07:34 +0100 Subject: [scikit-image] Release schedule In-Reply-To: References: <1482361925.3134024.826460969.50EBCD6F@webmail.messagingengine.com> Message-ID: <96ea005c-0765-4b4e-a1c3-b4e4adda4676@sciunto.org> Dear Developers, > > - Feature freeze 31 December 2016 > - Tag pre-relerase 15 January 2017 > - Tag release 20 January 2017 > > > To the core team: could someone, please, dedicate some time before the > freeze date to review and merge as many PR from the 0.13 roadmap as > possible (as I remember most of them are already finished in terms of > code)? > I would like to kindly rise again Egor's comment. We need more pairs of eyes on the pending PRs and pending issues, especially those tagged 0.13. These last days, I tried to go through them to check the status and re-tag them appropriately when needed. The Feature Freeze is in few days and we have PRs waiting in the stack that must be at least considered for 0.13. Thanks! -- Fran?ois Boulogne. http://www.sciunto.org GPG: 32D5F22F From winecoding at gmail.com Mon Dec 26 17:27:14 2016 From: winecoding at gmail.com (wine lover) Date: Mon, 26 Dec 2016 16:27:14 -0600 Subject: [scikit-image] image type issue and failed conversion Message-ID: Dear All, I was trying to use the above code segment for performing Contrast Limited Adaptive Histogram Equalization (CLAHE). def clahe_equalized(imgs): imgs_equalized = np.empty(imgs.shape) for i in range(imgs.shape[0]): print('imgs[i,0] ',imgs[i,0].dtype) print('imgs[i,0] ',imgs[i,0].max()) print('imgs[i,0] ',imgs[i,0].min()) imgs_equalized[i,0] = exposure.equalize_adapthist(imgs[i,0],clip_limit=0.03) return imgs_equalized The dtype is float64, maximum value is 255.0 and minimum value is 0.0 Running the program generates the following error message ( I only keep the related ones) imgs_equalized[i,0] = exposure.equalize_adapthist(imgs[i,0],clip_limit=0.03) raise ValueError("Images of type float must be between -1 and 1.") ValueError: Images of type float must be between -1 and 1. In accordance with the above error message and image characteristics, what are the best way to handle this scenario. I have been thinking of two approaches 1. add imgs[i,0] = imgs[i,0]/255. which scale it to 0 and 1 2. convert imgs[i,0] from float64 to unit8 but imgs[i,0] = imgs[i,0].astype(np.unit8) gives the error message such as imgs[i,0]=imgs[i,0].astype(np.unit8) AttributeError: 'module' object has no attribute 'unit8' Would you like to give any advice on this problem? Thank you very much! -------------- next part -------------- An HTML attachment was scrubbed... URL: From winecoding at gmail.com Mon Dec 26 18:22:25 2016 From: winecoding at gmail.com (wine lover) Date: Mon, 26 Dec 2016 17:22:25 -0600 Subject: [scikit-image] regarding the parameter of clip_limit for applying contrast limited adaptive historgram equalization Message-ID: Dear All, The following is an example given in opencv regarding applying Contrast Limited Adaptive Histogram Equalization (CLAHE) *import numpy as np* *import cv2* *img = cv2.imread('tsukuba_l.png',0)* *clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8,8))* *cl1 = clahe.apply(img)* Here the parameter clipLimit =2.0 In Skimage, CLAHE is perfored using *exposure.equalize_adapthist* For instance, in this example, http://scikit-image.org/docs/dev/auto_examples/plot_equalize.html *img_adapteq = exposure.equalize_adapthist(img, clip_limit=0.03)* My question is that how to setup the clip_limit value in skimage for a corresponding case in opencv For instance, in an example implemented using opencv, clipLimit is setup as 2.0; if I want to convert this implementation using skimage which value should I assign to clip_limit? According to the document looks like clip_limit between 0 and 1. *clip_limit : float, optional* *Clipping limit, normalized between 0 and 1 (higher values give more contrast).* while opencv does not have this limitation for clipLimit Thanks, Yuanyuan -------------- next part -------------- An HTML attachment was scrubbed... URL: From egor.v.panfilov at gmail.com Tue Dec 27 02:39:04 2016 From: egor.v.panfilov at gmail.com (Egor Panfilov) Date: Tue, 27 Dec 2016 10:39:04 +0300 Subject: [scikit-image] regarding the parameter of clip_limit for applying contrast limited adaptive historgram equalization In-Reply-To: References: Message-ID: Dear Yuanyuan, There is no strict correspondence between these two clip limits. If you would like to have something like OpenCV implementation of CLAHE, consider trying https://github.com/anntzer/clahe. Also, feel free to join the discussion in https://github.com/scikit-i mage/scikit-image/issues/2219. There you might find a bit more details. Regards, Egor 2016-12-27 2:22 GMT+03:00 wine lover : > Dear All, > > The following is an example given in opencv regarding applying Contrast > Limited Adaptive Histogram Equalization (CLAHE) > > *import numpy as np* > *import cv2* > *img = cv2.imread('tsukuba_l.png',0)* > *clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8,8))* > *cl1 = clahe.apply(img)* > > Here the parameter clipLimit =2.0 > > In Skimage, CLAHE is perfored using *exposure.equalize_adapthist* > > For instance, in this example, http://scikit-image.org/docs/ > dev/auto_examples/plot_equalize.html > > *img_adapteq = exposure.equalize_adapthist(img, clip_limit=0.03)* > > My question is that how to setup the clip_limit value in skimage for a > corresponding case in opencv > > > For instance, in an example implemented using opencv, clipLimit is setup > as 2.0; if I want to convert this implementation using skimage > which value should I assign to clip_limit? > > According to the document looks like clip_limit between 0 and 1. > *clip_limit : float, optional* > *Clipping limit, normalized between 0 and 1 (higher values give more > contrast).* > > while opencv does not have this limitation for clipLimit > > Thanks, > Yuanyuan > > _______________________________________________ > scikit-image mailing list > scikit-image at python.org > https://mail.python.org/mailman/listinfo/scikit-image > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From egor.v.panfilov at gmail.com Tue Dec 27 02:39:29 2016 From: egor.v.panfilov at gmail.com (Egor Panfilov) Date: Tue, 27 Dec 2016 10:39:29 +0300 Subject: [scikit-image] image type issue and failed conversion In-Reply-To: References: Message-ID: Dear Yuanyuan, First of all, it is not a good idea to initialize the array with values using `np.empty`. I'd recommend to use either `np.random.rand` or `np.random.randint`. As for main point of your question, I believe you might need http://scikit-image.org/docs/dev/api/skimage.html#img-as-float (see also http://scikit-image.org/docs/dev/user_guide/data_types.html ). So, you can either create an array of floats [0:1) via `np.random.rand`, or create an array of uints via `np.random.randint`, and call `img_as_float`. Then `equalize_adapthist` should work flawlessly. Regards, Egor 2016-12-27 1:27 GMT+03:00 wine lover : > Dear All, > > I was trying to use the above code segment for performing Contrast Limited > Adaptive Histogram Equalization (CLAHE). > def clahe_equalized(imgs): > imgs_equalized = np.empty(imgs.shape) > for i in range(imgs.shape[0]): > > print('imgs[i,0] ',imgs[i,0].dtype) > print('imgs[i,0] ',imgs[i,0].max()) > print('imgs[i,0] ',imgs[i,0].min()) > imgs_equalized[i,0] = exposure.equalize_adapthist( > imgs[i,0],clip_limit=0.03) > return imgs_equalized > > The dtype is float64, maximum value is 255.0 and minimum value is 0.0 > > Running the program generates the following error message ( I only > keep the related ones) > > imgs_equalized[i,0] = exposure.equalize_adapthist( > imgs[i,0],clip_limit=0.03) > raise ValueError("Images of type float must be between -1 and 1.") > ValueError: Images of type float must be between -1 and 1. > > In accordance with the above error message and image characteristics, what > are the best way to handle this scenario. > > I have been thinking of two approaches > > > 1. add imgs[i,0] = imgs[i,0]/255. which scale it to 0 and 1 > 2. convert imgs[i,0] from float64 to unit8 > > but imgs[i,0] = imgs[i,0].astype(np.unit8) gives the error message such as > imgs[i,0]=imgs[i,0].astype(np.unit8) > > AttributeError: 'module' object has no attribute 'unit8' > > Would you like to give any advice on this problem? Thank you very much! > > > > _______________________________________________ > scikit-image mailing list > scikit-image at python.org > https://mail.python.org/mailman/listinfo/scikit-image > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jni.soma at gmail.com Tue Dec 27 19:13:55 2016 From: jni.soma at gmail.com (Juan Nunez-Iglesias) Date: Wed, 28 Dec 2016 11:13:55 +1100 Subject: [scikit-image] image type issue and failed conversion In-Reply-To: References: <83a74af2-f12f-48c8-b942-fc0d0111058e@Spark> Message-ID: <7a592d04-e679-420d-a57d-116bc3a34abc@Spark> Oh, right, sorry, now I see what you're doing. Arrays are homogeneous, meaning every value has the same type. If you write: imgs[i, 0] = imgs[i, 0].astype(np.uint8) you are not changing the type of imgs, so you explicitly cast to uint8 and then the assignment (=) implicitly casts it back to float64. Oops! =) Please follow the advice of Egor and find the img_as_ubyte and img_as_float methods, and use those to convert images of different types. Juan. On 28 Dec 2016, 2:48 AM +1100, wine lover , wrote: > Hi Juan, > > Thanks for pointing the typo. I corrected it, and looks like imgs[i,0]=imgs[i,0].astype(np.unit8) does not solve the problem. > > Here is the screenshot of result > > imgs[i,0] ?(584, 565) > imgs[i,0] ?float64 > imgs[i,0] ?255.0 > imgs[i,0] ?0.0 > afte applying astype > imgs[i,0] ?(584, 565) > imgs[i,0] ?float64 > imgs[i,0] ?255.0 > imgs[i,0] ?0.0 > > Thanks, > Yuanyuan > > > > > On Tue, Dec 27, 2016 at 12:10 AM, Juan Nunez-Iglesias wrote: > > > Typo: unit8 -> uint8 > > > > > > > > > On 27 Dec 2016, 9:27 AM +1100, wine lover , wrote: > > > > Dear All, > > > > > > > > I was trying to use the above code segment for performing Contrast Limited Adaptive Histogram Equalization (CLAHE). > > > > def clahe_equalized(imgs): > > > > ? ? imgs_equalized = np.empty(imgs.shape) > > > > ? ? for i in range(imgs.shape[0]): > > > > > > > > ? ? ? ? ?print('imgs[i,0] ',imgs[i,0].dtype) > > > > ? ? ? ? ?print('imgs[i,0] ',imgs[i,0].max()) > > > > ? ? ? ? ?print('imgs[i,0] ',imgs[i,0].min()) > > > > ? ? ? ? ?imgs_equalized[i,0] = exposure.equalize_adapthist(imgs[i,0],clip_limit=0.03) > > > > ? ? return imgs_equalized > > > > > > > > The dtype is float64, maximum value is 255.0 and minimum value is 0.0 > > > > > > > > Running the program generates the following error message ( I only keep?the related ones) > > > > > > > > imgs_equalized[i,0] = exposure.equalize_adapthist(imgs[i,0],clip_limit=0.03) > > > > ? ?raise ValueError("Images of type float must be between -1 and 1.") > > > > ValueError: Images of type float must be between -1 and 1. > > > > > > > > In accordance with the above error message and image characteristics, what are the best way to handle this scenario. > > > > > > > > I have been thinking of two approaches > > > > > > > > > > > > - add imgs[i,0] = imgs[i,0]/255. ? which scale it to 0 and 1 > > > > - ?convert imgs[i,0] from float64 to unit8 > > > > > > > > but imgs[i,0] = imgs[i,0].astype(np.unit8) gives the error message such as > > > > ?imgs[i,0]=imgs[i,0].astype(np.unit8) > > > > AttributeError: 'module' object has no attribute 'unit8' > > > > > > > > Would you like to give any advice on this problem? Thank you very much! > > > > > > > > > > > > _______________________________________________ > > > > scikit-image mailing list > > > > scikit-image at python.org > > > > https://mail.python.org/mailman/listinfo/scikit-image > On 27 Dec 2016, 9:27 AM +1100, wine lover , wrote: > Dear All, > > I was trying to use the above code segment for performing Contrast Limited Adaptive Histogram Equalization (CLAHE). > def clahe_equalized(imgs): > ? ? imgs_equalized = np.empty(imgs.shape) > ? ? for i in range(imgs.shape[0]): > > ? ? ? ? ?print('imgs[i,0] ',imgs[i,0].dtype) > ? ? ? ? ?print('imgs[i,0] ',imgs[i,0].max()) > ? ? ? ? ?print('imgs[i,0] ',imgs[i,0].min()) > ? ? ? ? ?imgs_equalized[i,0] = exposure.equalize_adapthist(imgs[i,0],clip_limit=0.03) > ? ? return imgs_equalized > > The dtype is float64, maximum value is 255.0 and minimum value is 0.0 > > Running the program generates the following error message ( I only keep?the related ones) > > imgs_equalized[i,0] = exposure.equalize_adapthist(imgs[i,0],clip_limit=0.03) > ? ?raise ValueError("Images of type float must be between -1 and 1.") > ValueError: Images of type float must be between -1 and 1. > > In accordance with the above error message and image characteristics, what are the best way to handle this scenario. > > I have been thinking of two approaches > > > - add imgs[i,0] = imgs[i,0]/255. ? which scale it to 0 and 1 > - ?convert imgs[i,0] from float64 to unit8 > > but imgs[i,0] = imgs[i,0].astype(np.unit8) gives the error message such as > ?imgs[i,0]=imgs[i,0].astype(np.unit8) > AttributeError: 'module' object has no attribute 'unit8' > > Would you like to give any advice on this problem? Thank you very much! > > > _______________________________________________ > scikit-image mailing list > scikit-image at python.org > https://mail.python.org/mailman/listinfo/scikit-image -------------- next part -------------- An HTML attachment was scrubbed... URL: From simone.codeluppi at gmail.com Wed Dec 28 13:07:39 2016 From: simone.codeluppi at gmail.com (simone codeluppi) Date: Wed, 28 Dec 2016 18:07:39 +0000 Subject: [scikit-image] Image analysis pipeline improvement suggestions Message-ID: Hi all! I would like to pick your brain for some suggestion on how to modify my image analysis pipeline. I am analyzing terabytes of image stacks generated using a microscope. The current code I generated rely heavily on scikit-image, numpy and scipy. In order to speed up the analysis the code runs on a HPC computer ( https://www.nsc.liu.se/systems/triolith/) with MPI (mpi4py) for parallelization and hdf5 (h5py) for file storage. The development cycle of the code has been pretty painful mainly due to my non familiarity with mpi and problems in compiling parallel hdf5 (with many open/closing bugs). However, the big drawback is that each core has only 2Gb of RAM (no shared ram across nodes) and in order to run some of the processing steps i ended up reserving one node (16 cores) but running only 3 cores in order to have enough ram (image chunking won?t work in this case). As you can imagine this is extremely inefficient and i end up getting low priority in the queue system. Our lab currently bought a new 4 nodes server with shared RAM running hadoop. My goal is to move the parallelization of the processing to dask. I tested it before in another system and works great. The drawback is that, if I understood correctly, parallel hdf5 works only with MPI (driver=?mpio?). Hdf5 gave me quite a bit of headache but works well in keeping a good structure of the data and i can save everything as numpy arrays?.very handy. If I will move to hadoop/dask what do you think will be a good solution for data storage? Do you have any additional suggestion that can improve the layout of the pipeline? Any help will be greatly appreciated. Simone -- *Bad as he is, the Devil may be abus'd,* *Be falsy charg'd, and causelesly accus'd,* *When men, unwilling to be blam'd alone,* *Shift off these Crimes on Him which are their* *Own* *Daniel Defoe* simone.codeluppi at gmail.com simone at codeluppi.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From nathan.faggian at gmail.com Wed Dec 28 17:37:54 2016 From: nathan.faggian at gmail.com (Nathan Faggian) Date: Thu, 29 Dec 2016 09:37:54 +1100 Subject: [scikit-image] Image analysis pipeline improvement suggestions In-Reply-To: References: Message-ID: Hi Simone, I have had a little experience with HDF5 and am interested to see where you go with this. I wonder if you could use "feather": https://github.com/wesm/feather There was a recent post from Wes McKinney about feather, which sparked my interest: http://wesmckinney.com/blog/high-perf-arrow-to-pandas/ Do you use HDF5 to store intermediates? if so, I would try storing intermediates to a file format like feather and then reducing to a HDF5 file at the end. The reduction should be IO bound and not dependent on RAM so would suit your cluster. If you need to read a large array then I think HDF5 supports that (for single write but multiple reads) without the need for MPI - so this could map well to a tool like distributed: http://distributed.readthedocs.io/en/latest/ Not sure this helps, there is an assumption (on my part) that your intermediate calculations are not terabytes in size. Good luck! Nathan On 29 December 2016 at 05:07, simone codeluppi wrote: > Hi all! > > I would like to pick your brain for some suggestion on how to modify my > image analysis pipeline. > > I am analyzing terabytes of image stacks generated using a microscope. The > current code I generated rely heavily on scikit-image, numpy and scipy. In > order to speed up the analysis the code runs on a HPC computer ( > https://www.nsc.liu.se/systems/triolith/) with MPI (mpi4py) for > parallelization and hdf5 (h5py) for file storage. The development cycle of > the code has been pretty painful mainly due to my non familiarity with mpi > and problems in compiling parallel hdf5 (with many open/closing bugs). > However, the big drawback is that each core has only 2Gb of RAM (no shared > ram across nodes) and in order to run some of the processing steps i ended > up reserving one node (16 cores) but running only 3 cores in order to have > enough ram (image chunking won?t work in this case). As you can imagine > this is extremely inefficient and i end up getting low priority in the > queue system. > > > Our lab currently bought a new 4 nodes server with shared RAM running > hadoop. My goal is to move the parallelization of the processing to dask. I > tested it before in another system and works great. The drawback is that, > if I understood correctly, parallel hdf5 works only with MPI > (driver=?mpio?). Hdf5 gave me quite a bit of headache but works well in > keeping a good structure of the data and i can save everything as numpy > arrays?.very handy. > > > If I will move to hadoop/dask what do you think will be a good solution > for data storage? Do you have any additional suggestion that can improve > the layout of the pipeline? Any help will be greatly appreciated. > > > Simone > -- > *Bad as he is, the Devil may be abus'd,* > *Be falsy charg'd, and causelesly accus'd,* > *When men, unwilling to be blam'd alone,* > *Shift off these Crimes on Him which are their* > *Own* > > *Daniel Defoe* > > simone.codeluppi at gmail.com > > simone at codeluppi.org > > > _______________________________________________ > scikit-image mailing list > scikit-image at python.org > https://mail.python.org/mailman/listinfo/scikit-image > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From egor.v.panfilov at gmail.com Thu Dec 29 05:16:21 2016 From: egor.v.panfilov at gmail.com (Egor Panfilov) Date: Thu, 29 Dec 2016 13:16:21 +0300 Subject: [scikit-image] image type issue and failed conversion In-Reply-To: References: Message-ID: Hi Yuanyuan, In your example the image data range is not being rescaled as it already has dtype float. `img_as_float` will rescale from [0:255] to [0:1] only if the dtype of input ndarray is of integer family (and, in your case, uint8). Take a look: In [3]: nd_int = np.random.randint(0, 255, (3, 3)) In [4]: nd_int Out[4]: array([[ 85, 15, 60], [225, 252, 32], [162, 173, 34]]) In [5]: nd_int = nd_int.astype(np.uint8) In [6]: skimage.img_as_float(nd_int) Out[6]: array([[ 0.33333333, 0.05882353, 0.23529412], [ 0.88235294, 0.98823529, 0.1254902 ], [ 0.63529412, 0.67843137, 0.13333333]]) Please, notice that if your data lies in a range [0:255], but the ndarray dtype is not uint8 (e.g. uint16, int8, etc), you will get different results. Regards, Egor 2016-12-27 19:12 GMT+03:00 wine lover : > Hi Egor, > > Thank you for the suggestion. This is how I modify the code > > imgs_equalized = np.random.rand(imgs.shape[0],imgs.shape[1],imgs.shape[2], > imgs.shape[3]) > for i in range(imgs.shape[0]): > print('imgs[i,0] ',imgs[i,0].shape) > print('imgs[i,0] ',imgs[i,0].dtype) > print('imgs[i,0] ',imgs[i,0].max()) > print('imgs[i,0] ',imgs[i,0].min()) > imgs[i,0]=img_as_float(imgs[i,0]) > print('afte applying astype') > print('imgs[i,0] ',imgs[i,0].shape) > print('imgs[i,0] ',imgs[i,0].dtype) > print('imgs[i,0] ',imgs[i,0].max()) > print('imgs[i,0] ',imgs[i,0].min()) > > the output is > > imgs[i,0] (584, 565) > imgs[i,0] float64 > imgs[i,0] 255.0 > imgs[i,0] 0.0 > afte applying astype > imgs[i,0] (584, 565) > imgs[i,0] float64 > imgs[i,0] 255.0 > imgs[i,0] 0.0 > > > Looks like it does not convert the image type as I expected, in specific, > the maximum value. > > Thanks, > Yuanyuan > > > > > > > On Tue, Dec 27, 2016 at 1:39 AM, Egor Panfilov > wrote: > >> Dear Yuanyuan, >> >> First of all, it is not a good idea to initialize the array with values >> using `np.empty`. I'd recommend to use either `np.random.rand` or >> `np.random.randint`. >> >> As for main point of your question, I believe you might need >> http://scikit-image.org/docs/dev/api/skimage.html#img-as-float (see also >> http://scikit-image.org/docs/dev/user_guide/data_types.html ). >> So, you can either create an array of floats [0:1) via `np.random.rand`, >> or create an array of uints via `np.random.randint`, and call >> `img_as_float`. Then `equalize_adapthist` should work flawlessly. >> >> Regards, >> Egor >> >> 2016-12-27 1:27 GMT+03:00 wine lover : >> >>> Dear All, >>> >>> I was trying to use the above code segment for performing Contrast >>> Limited Adaptive Histogram Equalization (CLAHE). >>> def clahe_equalized(imgs): >>> imgs_equalized = np.empty(imgs.shape) >>> for i in range(imgs.shape[0]): >>> >>> print('imgs[i,0] ',imgs[i,0].dtype) >>> print('imgs[i,0] ',imgs[i,0].max()) >>> print('imgs[i,0] ',imgs[i,0].min()) >>> imgs_equalized[i,0] = exposure.equalize_adapthist(im >>> gs[i,0],clip_limit=0.03) >>> return imgs_equalized >>> >>> The dtype is float64, maximum value is 255.0 and minimum value is 0.0 >>> >>> Running the program generates the following error message ( I only >>> keep the related ones) >>> >>> imgs_equalized[i,0] = exposure.equalize_adapthist(im >>> gs[i,0],clip_limit=0.03) >>> raise ValueError("Images of type float must be between -1 and 1.") >>> ValueError: Images of type float must be between -1 and 1. >>> >>> In accordance with the above error message and image characteristics, >>> what are the best way to handle this scenario. >>> >>> I have been thinking of two approaches >>> >>> >>> 1. add imgs[i,0] = imgs[i,0]/255. which scale it to 0 and 1 >>> 2. convert imgs[i,0] from float64 to unit8 >>> >>> but imgs[i,0] = imgs[i,0].astype(np.unit8) gives the error message such >>> as >>> imgs[i,0]=imgs[i,0].astype(np.unit8) >>> >>> AttributeError: 'module' object has no attribute 'unit8' >>> >>> Would you like to give any advice on this problem? Thank you very much! >>> >>> >>> >>> _______________________________________________ >>> scikit-image mailing list >>> scikit-image at python.org >>> https://mail.python.org/mailman/listinfo/scikit-image >>> >>> >> >> _______________________________________________ >> scikit-image mailing list >> scikit-image at python.org >> https://mail.python.org/mailman/listinfo/scikit-image >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From winecoding at gmail.com Thu Dec 29 09:28:54 2016 From: winecoding at gmail.com (wine lover) Date: Thu, 29 Dec 2016 08:28:54 -0600 Subject: [scikit-image] image type issue and failed conversion In-Reply-To: References: Message-ID: Hi Egor, Hi Juan, Thank you very much for the help! Yuanyuan On Thu, Dec 29, 2016 at 4:16 AM, Egor Panfilov wrote: > Hi Yuanyuan, > > In your example the image data range is not being rescaled as it already > has dtype float. `img_as_float` will rescale from [0:255] to [0:1] only > if the dtype of input ndarray is of integer family (and, in your case, > uint8). > > Take a look: > In [3]: nd_int = np.random.randint(0, 255, (3, 3)) > > In [4]: nd_int > Out[4]: > array([[ 85, 15, 60], > [225, 252, 32], > [162, 173, 34]]) > > In [5]: nd_int = nd_int.astype(np.uint8) > > In [6]: skimage.img_as_float(nd_int) > Out[6]: > array([[ 0.33333333, 0.05882353, 0.23529412], > [ 0.88235294, 0.98823529, 0.1254902 ], > [ 0.63529412, 0.67843137, 0.13333333]]) > > Please, notice that if your data lies in a range [0:255], but the ndarray > dtype is not uint8 (e.g. uint16, int8, etc), you will get different results. > > Regards, > Egor > > 2016-12-27 19:12 GMT+03:00 wine lover : > >> Hi Egor, >> >> Thank you for the suggestion. This is how I modify the code >> >> imgs_equalized = np.random.rand(imgs.shape[0],i >> mgs.shape[1],imgs.shape[2],imgs.shape[3]) >> for i in range(imgs.shape[0]): >> print('imgs[i,0] ',imgs[i,0].shape) >> print('imgs[i,0] ',imgs[i,0].dtype) >> print('imgs[i,0] ',imgs[i,0].max()) >> print('imgs[i,0] ',imgs[i,0].min()) >> imgs[i,0]=img_as_float(imgs[i,0]) >> print('afte applying astype') >> print('imgs[i,0] ',imgs[i,0].shape) >> print('imgs[i,0] ',imgs[i,0].dtype) >> print('imgs[i,0] ',imgs[i,0].max()) >> print('imgs[i,0] ',imgs[i,0].min()) >> >> the output is >> >> imgs[i,0] (584, 565) >> imgs[i,0] float64 >> imgs[i,0] 255.0 >> imgs[i,0] 0.0 >> afte applying astype >> imgs[i,0] (584, 565) >> imgs[i,0] float64 >> imgs[i,0] 255.0 >> imgs[i,0] 0.0 >> >> >> Looks like it does not convert the image type as I expected, in specific, >> the maximum value. >> >> Thanks, >> Yuanyuan >> >> >> >> >> >> >> On Tue, Dec 27, 2016 at 1:39 AM, Egor Panfilov > > wrote: >> >>> Dear Yuanyuan, >>> >>> First of all, it is not a good idea to initialize the array with values >>> using `np.empty`. I'd recommend to use either `np.random.rand` or >>> `np.random.randint`. >>> >>> As for main point of your question, I believe you might need >>> http://scikit-image.org/docs/dev/api/skimage.html#img-as-float (see >>> also http://scikit-image.org/docs/dev/user_guide/data_types.html ). >>> So, you can either create an array of floats [0:1) via `np.random.rand`, >>> or create an array of uints via `np.random.randint`, and call >>> `img_as_float`. Then `equalize_adapthist` should work flawlessly. >>> >>> Regards, >>> Egor >>> >>> 2016-12-27 1:27 GMT+03:00 wine lover : >>> >>>> Dear All, >>>> >>>> I was trying to use the above code segment for performing Contrast >>>> Limited Adaptive Histogram Equalization (CLAHE). >>>> def clahe_equalized(imgs): >>>> imgs_equalized = np.empty(imgs.shape) >>>> for i in range(imgs.shape[0]): >>>> >>>> print('imgs[i,0] ',imgs[i,0].dtype) >>>> print('imgs[i,0] ',imgs[i,0].max()) >>>> print('imgs[i,0] ',imgs[i,0].min()) >>>> imgs_equalized[i,0] = exposure.equalize_adapthist(im >>>> gs[i,0],clip_limit=0.03) >>>> return imgs_equalized >>>> >>>> The dtype is float64, maximum value is 255.0 and minimum value is 0.0 >>>> >>>> Running the program generates the following error message ( I only >>>> keep the related ones) >>>> >>>> imgs_equalized[i,0] = exposure.equalize_adapthist(im >>>> gs[i,0],clip_limit=0.03) >>>> raise ValueError("Images of type float must be between -1 and 1.") >>>> ValueError: Images of type float must be between -1 and 1. >>>> >>>> In accordance with the above error message and image characteristics, >>>> what are the best way to handle this scenario. >>>> >>>> I have been thinking of two approaches >>>> >>>> >>>> 1. add imgs[i,0] = imgs[i,0]/255. which scale it to 0 and 1 >>>> 2. convert imgs[i,0] from float64 to unit8 >>>> >>>> but imgs[i,0] = imgs[i,0].astype(np.unit8) gives the error message such >>>> as >>>> imgs[i,0]=imgs[i,0].astype(np.unit8) >>>> >>>> AttributeError: 'module' object has no attribute 'unit8' >>>> >>>> Would you like to give any advice on this problem? Thank you very much! >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> scikit-image mailing list >>>> scikit-image at python.org >>>> https://mail.python.org/mailman/listinfo/scikit-image >>>> >>>> >>> >>> _______________________________________________ >>> scikit-image mailing list >>> scikit-image at python.org >>> https://mail.python.org/mailman/listinfo/scikit-image >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fboulogne at sciunto.org Fri Dec 30 07:26:16 2016 From: fboulogne at sciunto.org (=?UTF-8?Q?Fran=c3=a7ois_Boulogne?=) Date: Fri, 30 Dec 2016 13:26:16 +0100 Subject: [scikit-image] Future of nose and issues for scikit-image Message-ID: Dear devs, I would like to raise a potential issue with our test suite. Let me list the facts. Our tests rely on nose. As you may have seen, we often have deprecation warnings (See https://github.com/scikit-image/scikit-image/issues/2414 ). These warnings come from nose, https://github.com/nose-devs/nose/issues/929 a ticket opened in Jun '15. Tom Caswell opened a PR in Sept '15 https://github.com/nose-devs/nose/pull/952 that got only little attention. In another of our issues, some tests are not run (those in directories starting by an underscore). https://github.com/scikit-image/scikit-image/issues/2127 As stated in our issue, it is hard-coded in nose and WON'T be fixed. Upstream suggests to upgrade to nose2. Egor investigated this possibility but it sounds not possible (Egor, can you elaborate?) It seems clear that nose is not maintained any more (last commit, 10 months ago, last release June '15). The deprecation warning we have will turn to a broken code with python 3.6 that has just been released. I let you check that my interpretation is correct, but to me, it represents a major issue for scikit-image. Feel free to comment and to correct me if necessary. I'm not well-experienced with the different libraries for unittesting. Best, -- Fran?ois Boulogne. http://www.sciunto.org GPG: 32D5F22F From nelle.varoquaux at gmail.com Fri Dec 30 08:36:28 2016 From: nelle.varoquaux at gmail.com (Nelle Varoquaux) Date: Fri, 30 Dec 2016 14:36:28 +0100 Subject: [scikit-image] Future of nose and issues for scikit-image In-Reply-To: References: Message-ID: Hello, Both matplotlib and sklearn are moving away from nose. I think sklearn's way of moving away is both very sustainable and the transition has been very smooth. The standard library unittest has been wrapped such that the API remains with nose/numpy.testing in a utils.testing module. That means the test suite needs very little modification, and we kept nose's very nice API. We still use nosetests as our test runner. If scikit-image decides to move forward into that direction, I'd be happy to help with the transition. Cheers, N On 30 December 2016 at 13:26, Fran?ois Boulogne wrote: > Dear devs, > > I would like to raise a potential issue with our test suite. Let me list > the facts. > > Our tests rely on nose. As you may have seen, we often have deprecation > warnings (See https://github.com/scikit-image/scikit-image/issues/2414 > ). These warnings come from nose, > https://github.com/nose-devs/nose/issues/929 a ticket opened in Jun '15. > Tom Caswell opened a PR in Sept '15 > https://github.com/nose-devs/nose/pull/952 that got only little attention. > > In another of our issues, some tests are not run (those in directories > starting by an underscore). > https://github.com/scikit-image/scikit-image/issues/2127 As stated in > our issue, it is hard-coded in nose and WON'T be fixed. Upstream > suggests to upgrade to nose2. Egor investigated this possibility but it > sounds not possible (Egor, can you elaborate?) > > It seems clear that nose is not maintained any more (last commit, 10 > months ago, last release June '15). The deprecation warning we have will > turn to a broken code with python 3.6 that has just been released. > > I let you check that my interpretation is correct, but to me, it > represents a major issue for scikit-image. Feel free to comment and to > correct me if necessary. I'm not well-experienced with the different > libraries for unittesting. > > Best, > > -- > Fran?ois Boulogne. > http://www.sciunto.org > GPG: 32D5F22F > > > _______________________________________________ > scikit-image mailing list > scikit-image at python.org > https://mail.python.org/mailman/listinfo/scikit-image From fboulogne at sciunto.org Sat Dec 31 05:42:29 2016 From: fboulogne at sciunto.org (=?UTF-8?Q?Fran=c3=a7ois_Boulogne?=) Date: Sat, 31 Dec 2016 11:42:29 +0100 Subject: [scikit-image] Future of nose and issues for scikit-image In-Reply-To: References: Message-ID: <08c990bf-f2f0-ab5e-4063-f4300e46900f@sciunto.org> Hello Nelle, Thank you for your nice proposition, it's really appreciated. To me, this sounds like the way to move but let's wait for other reactions. Best, -- Fran?ois Boulogne. http://www.sciunto.org GPG: 32D5F22F From stefanv at berkeley.edu Sat Dec 31 14:29:55 2016 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Sat, 31 Dec 2016 11:29:55 -0800 Subject: [scikit-image] Future of nose and issues for scikit-image In-Reply-To: <08c990bf-f2f0-ab5e-4063-f4300e46900f@sciunto.org> References: <08c990bf-f2f0-ab5e-4063-f4300e46900f@sciunto.org> Message-ID: <159565c1f38.273e.acf34a9c767d7bb498a799333be0433e@fastmail.com> Hi Fran?ois Thanks for bringing this up. I've now started using pytest for new projects, and I think it's possible to have a test suite that runs under both that and nose. However, I'm not sure if such backward compatibility gives us significant advantage, and suggest we make pytest the recommended and default way of running the test suite. Best regards St?fan On December 31, 2016 02:43:04 Fran?ois Boulogne wrote: > Hello Nelle, > > Thank you for your nice proposition, it's really appreciated. To me, > this sounds like the way to move but let's wait for other reactions. > > Best, > > -- > Fran?ois Boulogne. > http://www.sciunto.org > GPG: 32D5F22F > > > _______________________________________________ > scikit-image mailing list > scikit-image at python.org > https://mail.python.org/mailman/listinfo/scikit-image From steven.silvester at gmail.com Sat Dec 31 18:05:14 2016 From: steven.silvester at gmail.com (Steven Silvester) Date: Sat, 31 Dec 2016 17:05:14 -0600 Subject: [scikit-image] Future of nose and issues for scikit-image In-Reply-To: <159565c1f38.273e.acf34a9c767d7bb498a799333be0433e@fastmail.com> References: <08c990bf-f2f0-ab5e-4063-f4300e46900f@sciunto.org> <159565c1f38.273e.acf34a9c767d7bb498a799333be0433e@fastmail.com> Message-ID: <98FF2884-49FF-4421-A6C3-DE2B321A710E@gmail.com> Hi all, I agree with St?fan that a forward-looking clean break would be best. It looks like a rather large undertaking though, judging from https://github.com/matplotlib/matplotlib/pull/5405 . Regards, Steve > On Dec 31, 2016, at 1:29 PM, Stefan van der Walt wrote: > > Hi Fran?ois > > Thanks for bringing this up. I've now started using pytest for new projects, and I think it's possible to have a test suite that runs under both that and nose. However, I'm not sure if such backward compatibility gives us significant advantage, and suggest we make pytest the recommended and default way of running the test suite. > > Best regards > St?fan > > > On December 31, 2016 02:43:04 Fran?ois Boulogne wrote: > >> Hello Nelle, >> >> Thank you for your nice proposition, it's really appreciated. To me, >> this sounds like the way to move but let's wait for other reactions. >> >> Best, >> >> -- >> Fran?ois Boulogne. >> http://www.sciunto.org >> GPG: 32D5F22F >> >> >> _______________________________________________ >> scikit-image mailing list >> scikit-image at python.org >> https://mail.python.org/mailman/listinfo/scikit-image > > > _______________________________________________ > scikit-image mailing list > scikit-image at python.org > https://mail.python.org/mailman/listinfo/scikit-image -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefanv at berkeley.edu Sat Dec 31 18:26:06 2016 From: stefanv at berkeley.edu (Stefan van der Walt) Date: Sat, 31 Dec 2016 15:26:06 -0800 Subject: [scikit-image] Future of nose and issues for scikit-image In-Reply-To: <98FF2884-49FF-4421-A6C3-DE2B321A710E@gmail.com> References: <08c990bf-f2f0-ab5e-4063-f4300e46900f@sciunto.org> <159565c1f38.273e.acf34a9c767d7bb498a799333be0433e@fastmail.com> <98FF2884-49FF-4421-A6C3-DE2B321A710E@gmail.com> Message-ID: <1483226766.3931153.833987329.7617F174@webmail.messagingengine.com> On Sat, Dec 31, 2016, at 15:05, Steven Silvester wrote: > I agree with St?fan that a forward-looking clean break would be best. > It looks like a rather large undertaking though, judging from > https://github.com/matplotlib/matplotlib/pull/5405. Perhaps we are better off than matplotlib: $ git grep "from nose" skimage/_shared/testing.py: from nose.tools import assert_less skimage/_shared/testing.py: from nose.tools import assert_greater skimage/_shared/tests/test_testing.py:from nose.tools import (assert_true, assert_raises, assert_equal) skimage/draw/tests/test_draw3d.py:from nose.tools import raises skimage/io/tests/test_sift.py:from nose.tools import * skimage/novice/tests/test_novice.py:from nose.tools import assert_true skimage/segmentation/tests/test_quickshift.py:from nose.tools import assert_true skimage/util/tests/test_montage.py:from nose.tools import assert_equal, raises skimage/util/tests/test_shape.py:from nose.tools import raises St?fan -------------- next part -------------- An HTML attachment was scrubbed... URL: