From srutten at scannexus.nl Mon Jul 6 09:54:09 2020 From: srutten at scannexus.nl (Rutten, Sanne (SCANNEXUS)) Date: Mon, 6 Jul 2020 13:54:09 +0000 Subject: [Neuroimaging] [nipype] known problems running nipype in macOS Catalina? Message-ID: <1C4B9060-227E-4164-94F9-B7CCD2C6C803@scannexus.nl> Dear all, I?m considering upgrading my macOS to Catalina, and I was wondering whether any one experienced incompatibility issues while doing this? Thanks in advance. Best wishes, Sanne Rutten From satra at mit.edu Mon Jul 6 10:26:50 2020 From: satra at mit.edu (Satrajit Ghosh) Date: Mon, 6 Jul 2020 10:26:50 -0400 Subject: [Neuroimaging] [nipype] known problems running nipype in macOS Catalina? In-Reply-To: <1C4B9060-227E-4164-94F9-B7CCD2C6C803@scannexus.nl> References: <1C4B9060-227E-4164-94F9-B7CCD2C6C803@scannexus.nl> Message-ID: hi, there are no known problems with nipype, which is itself just a python software. however, depending on which version you are upgrading from certain underlying software that nipype calls may require some additional hand holding for general installation. this is because catalina changes which parts of the filesystem a user has write access to. depending on how you use os x, you may have to shift around locations of software. however, we have not heard any issues with using nipype and other neuroimaging software in catalina. cheers, satra On Mon, Jul 6, 2020 at 9:58 AM Rutten, Sanne (SCANNEXUS) < srutten at scannexus.nl> wrote: > Dear all, > > I?m considering upgrading my macOS to Catalina, and I was wondering > whether any one experienced incompatibility issues while doing this? > > Thanks in advance. > > Best wishes, Sanne Rutten > > > > _______________________________________________ > Neuroimaging mailing list > Neuroimaging at python.org > https://mail.python.org/mailman/listinfo/neuroimaging > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.puzniak at med.ovgu.de Wed Jul 15 05:47:35 2020 From: robert.puzniak at med.ovgu.de (Puzniak, Robert) Date: Wed, 15 Jul 2020 09:47:35 +0000 Subject: [Neuroimaging] Transformation of images into common space Message-ID: Hello everybody, I have a following problem - I have 2 different 3D images (different images sizes, different voxel sizes, different order of axes; both have sform and qforms defined), and I would like to transform the in a way, that they will be stored in arrays of the same shapes, with matching voxel coordinates (so a given voxel in scanner space will have the same voxel coordinates). I suppose the affine transformations are a key here, but I'm unsure how to proceed. Any hints will be much appreciated. Thank You, Robert Puzniak -- M.Sc. Robert Puzniak Marie Curie Fellow - NextGenVis Network Visual Processing Lab, Ophthalmic Dpt. Otto-von-Guericke-Universitaet Leipziger Str. 44 39120 Magdeburg, Germany ++49 (0)391 67 21722 From elef at indiana.edu Fri Jul 17 09:53:43 2020 From: elef at indiana.edu (Eleftherios Garyfallidis) Date: Fri, 17 Jul 2020 09:53:43 -0400 Subject: [Neuroimaging] Transformation of images into common space In-Reply-To: References: Message-ID: Robert, have you read this? https://dipy.org/documentation/1.1.1./examples_built/affine_registration_3d/#example-affine-registration-3d Best, Eleftherios On Fri, Jul 17, 2020 at 8:03 AM Puzniak, Robert wrote: > Hello everybody, > > I have a following problem - I have 2 different 3D images (different > images sizes, different voxel sizes, different order of axes; both have > sform and qforms defined), and I would like to transform the in a way, that > they will be stored in arrays of the same shapes, with matching voxel > coordinates (so a given voxel in scanner space will have the same voxel > coordinates). I suppose the affine transformations are a key here, but I'm > unsure how to proceed. Any hints will be much appreciated. > > Thank You, > > > Robert Puzniak > > > -- > M.Sc. Robert Puzniak > Marie Curie Fellow - NextGenVis Network > > Visual Processing Lab, Ophthalmic Dpt. > Otto-von-Guericke-Universitaet > Leipziger Str. 44 > 39120 Magdeburg, Germany > > ++49 (0)391 67 21722 > _______________________________________________ > Neuroimaging mailing list > Neuroimaging at python.org > https://mail.python.org/mailman/listinfo/neuroimaging > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Fri Jul 17 10:05:36 2020 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 17 Jul 2020 15:05:36 +0100 Subject: [Neuroimaging] Transformation of images into common space In-Reply-To: References: Message-ID: Hi, On Fri, Jul 17, 2020 at 1:04 PM Puzniak, Robert wrote: > > Hello everybody, > > I have a following problem - I have 2 different 3D images (different images sizes, different voxel sizes, different order of axes; both have sform and qforms defined), and I would like to transform the in a way, that they will be stored in arrays of the same shapes, with matching voxel coordinates (so a given voxel in scanner space will have the same voxel coordinates). I suppose the affine transformations are a key here, but I'm unsure how to proceed. Any hints will be much appreciated. I'm guessing you mean that the images are already in the same world-space (they are already registered), and you want to resample the images into the same voxel space? Have a look at: https://nipy.org/nibabel/reference/nibabel.processing.html#nibabel.processing.resample_from_to Does it help? Cheers, Matthew From rolandomasisobando at gmail.com Fri Jul 17 18:57:18 2020 From: rolandomasisobando at gmail.com (Rolando Masis-Obando) Date: Fri, 17 Jul 2020 18:57:18 -0400 Subject: [Neuroimaging] (1) How to set bilateral min values for colorbar in pysurfer? and (2) How to ignore nans when plotting a brain with add_data or add_overaly? Message-ID: Hi everyone! I hope everything is going well for everyone. I have two questions, the first is: I?ve been trying to figure out how to set up a threshold value in pysurfer using either add_overlay (with min) or add_date (with threshold) for both hemispheres. It turns out, I can?t seem to set a universal threshold that works with both hemispheres. For example, let?s play with this example. I have a left and right hemispheres in fs6 space under a 40962 array. The first thing I do is find the minimum and maximum values of each hemisphere and in order to keep the colorbars consistent across both hemispheres, I want to set a constant minimum value and a constant maximum value. let?s say that: left_min = -.2065 right_min = -.1381 left_max = .1664 right_max = .1422 My question is, how can I use pysurfer to use the same colorbar min and max values? When I try to do this, by picking left_min and left_max for both hemispheres like so: brain.add_overlay(brain_data, min=left_min, max=left_max, hemi='lh') brain.add_overlay(brain_data, min=left_min, max=left_max, hemi='rh') OR with add_date like so (with *threshold* either set to None or to left_min): brain.add_data(brain_data[:,np.newaxis],time_label = None,hemi='lh',min=left_min, max=left_max, thresh=thresh) brain.add_data(brain_data[:,np.newaxis],time_label = None,hemi='rh',min=left_min, max=left_max, thresh=thresh) I get the following error: TraitError: The 'lower_threshold' trait of a Threshold instance must be -0.1381019441953028 <= a number <= 0.14220765405834088, but a value of -0.20656023097769433 was specified. I used to be able to do this with no problem in an older version of pysurfer, but it seems I get this error with the new one. Any creative alternative solutions to this obstacle? My second question: Is there an easy way to set the transparency (alpha) of nan values easily in pysurfer, in other words, ignore nans? Sometimes when I plot a brain, the medial wall gets colored with a default dark red color, rather than showing the medial wall empty. I usually have the medial wall set to np.nan, and in an older version of pysurfer this was always ignored (I think). But, in this new version, it sometimes gets colored, hence why I have to use the minimum or threshold to avoid that problem. But, as can be seen from question 1, there are limits to this. Thanks in advance! Best, Rolando -------------- next part -------------- An HTML attachment was scrubbed... URL: From bjarnijo90 at gmail.com Wed Jul 22 04:55:01 2020 From: bjarnijo90 at gmail.com (Bjarni J) Date: Wed, 22 Jul 2020 10:55:01 +0200 Subject: [Neuroimaging] Can it be used to make npy files from dcom? Message-ID: Hi can nibabel be used to create npy files from a dicom image? I have a bunch of dicom's that I want converted to numpy arrays but I can't seem to find a good solution. -- Mbk./Mvh./Regards, Bjarni J -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Wed Jul 22 05:16:00 2020 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 22 Jul 2020 10:16:00 +0100 Subject: [Neuroimaging] Can it be used to make npy files from dcom? In-Reply-To: References: Message-ID: Hi, On Wed, Jul 22, 2020 at 9:58 AM Bjarni J wrote: > > Hi can nibabel be used to create npy files from a dicom image? I have a bunch of dicom's that I want converted to numpy arrays but I can't seem to find a good solution. > Can you just use the pydicom package for this? [ins] In [1]: import pydicom [ins] In [2]: dcm = pydicom.read_file('1.dcm') [ins] In [3]: arr = dcm.pixel_array [ins] In [4]: arr Out[4]: array([[ 0, 1, 2, ..., 253, 254, 255], [ 256, 257, 258, ..., 509, 510, 511], [ 512, 513, 514, ..., 765, 766, 767], ..., [3328, 3329, 3330, ..., 3581, 3582, 3583], [3584, 3585, 3586, ..., 3837, 3838, 3839], [3840, 3841, 3842, ..., 4093, 4094, 4095]], dtype=uint16) Cheers, Matthew From matthew.brett at gmail.com Thu Jul 30 10:53:28 2020 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 30 Jul 2020 15:53:28 +0100 Subject: [Neuroimaging] CZI grant - what would you like to see in Nibabel? In-Reply-To: References: <20200728210236.dqmjjl6s5uruvrbm@phare.normalesup.org> <20200729161235.uezevmmxhrd267xk@phare.normalesup.org> Message-ID: Hi, Although this is pushing things close the wire, I am currently trying to work out whether to apply for a Chan Zuckerberg grant for Nibabel: https://chanzuckerberg.com/wp-content/uploads/2020/06/EOSS-3-RFA-Announcement-and-Instructions-CYCLE-3-COMBINED-1.pdf Before we do that, I wanted to ask - if y'all could get anything into Nibabel - what would it be? So far, in various conversations, Chris Markiewicz and I came up with: * More comprehensive DICOM to NIfTI etc support * Labelled axes for images, maybe via xarray * Integrated API for surface formats [1] * HDF5 serialization for many image formats * Ability to read images directly from URLs, read / write images to / from AWS, GCS, Azure data storage addresses [2] * Optimized visualization for web and desktop Is there anything else you'd like to see? See you, Matthew [1] https://github.com/nipy/nibabel/issues/936 [2] c.f. https://pandas.pydata.org/pandas-docs/version/1.1.0/whatsnew/v1.1.0.html#fsspec-now-used-for-filesystem-handling From thomas.tv.vincent at gmail.com Thu Jul 30 11:27:56 2020 From: thomas.tv.vincent at gmail.com (thomas VINCENT) Date: Thu, 30 Jul 2020 11:27:56 -0400 Subject: [Neuroimaging] CZI grant - what would you like to see in Nibabel? In-Reply-To: References: <20200728210236.dqmjjl6s5uruvrbm@phare.normalesup.org> <20200729161235.uezevmmxhrd267xk@phare.normalesup.org> Message-ID: Hello, I think providing tools for multimodality would be great. The basic experimental use case is that we have several simultaneous imaging measures, eg fmri eeg and breathing. The first concern is to have a common container for neuroimaging data from different modalities. The second one is to conveniently handle temporal synchronization between measures and then interpolation on the same temporal grid. My one cent. Thanks for all the great work on nibabel so far! Thomas. Le jeu. 30 juill. 2020 11 h 01, Matthew Brett a ?crit : > Hi, > > Although this is pushing things close the wire, I am currently trying > to work out whether to apply for a Chan Zuckerberg grant for Nibabel: > > > https://chanzuckerberg.com/wp-content/uploads/2020/06/EOSS-3-RFA-Announcement-and-Instructions-CYCLE-3-COMBINED-1.pdf > > Before we do that, I wanted to ask - if y'all could get anything into > Nibabel - what > would it be? > > So far, in various conversations, Chris Markiewicz and I came up with: > > * More comprehensive DICOM to NIfTI etc support > * Labelled axes for images, maybe via xarray > * Integrated API for surface formats [1] > * HDF5 serialization for many image formats > * Ability to read images directly from URLs, read / write images to / > from AWS, GCS, Azure data storage addresses [2] > * Optimized visualization for web and desktop > > Is there anything else you'd like to see? > > See you, > > Matthew > > [1] https://github.com/nipy/nibabel/issues/936 > [2] c.f. > https://pandas.pydata.org/pandas-docs/version/1.1.0/whatsnew/v1.1.0.html#fsspec-now-used-for-filesystem-handling > _______________________________________________ > Neuroimaging mailing list > Neuroimaging at python.org > https://mail.python.org/mailman/listinfo/neuroimaging > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Thu Jul 30 11:39:27 2020 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 30 Jul 2020 16:39:27 +0100 Subject: [Neuroimaging] CZI grant - what would you like to see in Nibabel? In-Reply-To: References: <20200728210236.dqmjjl6s5uruvrbm@phare.normalesup.org> <20200729161235.uezevmmxhrd267xk@phare.normalesup.org> Message-ID: Hi, On Thu, Jul 30, 2020 at 4:28 PM thomas VINCENT wrote: > > Hello, > > I think providing tools for multimodality would be great. The basic experimental use case is that we have several simultaneous imaging measures, eg fmri eeg and breathing. The first concern is to have a common container for neuroimaging data from different modalities. The second one is to conveniently handle temporal synchronization between measures and then interpolation on the same temporal grid. > > My one cent. Excellent - thank you. I've taken the liberty of putting this up as an issue, I hope that's OK. https://github.com/nipy/nibabel/issues/941 Please do ask me to close if it's not OK. I've asked for more details in the issue, if you have time. Cheers, Matthew From olivetti at fbk.eu Thu Jul 30 11:59:49 2020 From: olivetti at fbk.eu (Emanuele Olivetti) Date: Thu, 30 Jul 2020 17:59:49 +0200 Subject: [Neuroimaging] CZI grant - what would you like to see in Nibabel? In-Reply-To: References: <20200728210236.dqmjjl6s5uruvrbm@phare.normalesup.org> <20200729161235.uezevmmxhrd267xk@phare.normalesup.org> Message-ID: Hi Matthew, My colleagues and I would like to have a (much) faster loader of tractography data with a smaller memory footprint, especially for TRK (Trackvis) files. Just to give you an idea - which, yes, it's an extreme case but definitely not so infrequent for us - when loading a very large tractogram of 10 million streamlines it takes: - 15 minutes and approximately 10Gb of RAM with Nibabel - 1 minute and approximately 5Gb of RAM with our custom TRK loader (it works but code style is meh, https://github.com/FBK-NILab/nilab/blob/master/load_trk.py) - 6 seconds and 3.5Gb of RAM if we previously save the tractogram as numpy array (numpy.savez()) and then load it with numpy.load(). But in this case, the streamlines are resampled to a fixed number of points. Thank you for all the work done! Emanuele On Thu, Jul 30, 2020 at 5:01 PM Matthew Brett wrote: > Hi, > > Although this is pushing things close the wire, I am currently trying > to work out whether to apply for a Chan Zuckerberg grant for Nibabel: > > > https://chanzuckerberg.com/wp-content/uploads/2020/06/EOSS-3-RFA-Announcement-and-Instructions-CYCLE-3-COMBINED-1.pdf > > Before we do that, I wanted to ask - if y'all could get anything into > Nibabel - what > would it be? > > So far, in various conversations, Chris Markiewicz and I came up with: > > * More comprehensive DICOM to NIfTI etc support > * Labelled axes for images, maybe via xarray > * Integrated API for surface formats [1] > * HDF5 serialization for many image formats > * Ability to read images directly from URLs, read / write images to / > from AWS, GCS, Azure data storage addresses [2] > * Optimized visualization for web and desktop > > Is there anything else you'd like to see? > > See you, > > Matthew > > [1] https://github.com/nipy/nibabel/issues/936 > [2] c.f. > https://pandas.pydata.org/pandas-docs/version/1.1.0/whatsnew/v1.1.0.html#fsspec-now-used-for-filesystem-handling > _______________________________________________ > Neuroimaging mailing list > Neuroimaging at python.org > https://mail.python.org/mailman/listinfo/neuroimaging > -- Emanuele Olivetti, Ph.D. Senior Research Scientist at Bruno Kessler Foundation (FBK-ICT) NeuroInformatics Laboratory (NILab) http://nilab.fbk.eu Center for Mind and Brain Sciences (CIMeC), University of Trento Via delle Regole 101 - 38123 Mattarello (Trento), ITALY olivetti at fbk.eu - +39 0461314179 emanuele.olivetti at unitn.it - +39 0461282760 -- _FBK vi invita a leggere il suo?Piano di rientro ? | FBK invites you to read its?Premises Reopening Plan ._ -- Le informazioni contenute nella presente comunicazione sono di natura? privata e come tali sono da considerarsi riservate ed indirizzate? esclusivamente ai destinatari indicati e per le finalit? strettamente? legate al relativo contenuto. Se avete ricevuto questo messaggio per? errore, vi preghiamo di eliminarlo e di inviare una comunicazione? all?indirizzo e-mail del mittente. -- The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you received this in error, please contact the sender and delete the material. -------------- next part -------------- An HTML attachment was scrubbed... URL: From frakkopesto at gmail.com Thu Jul 30 12:05:03 2020 From: frakkopesto at gmail.com (Franco Pestilli) Date: Thu, 30 Jul 2020 11:05:03 -0500 Subject: [Neuroimaging] CZI grant - what would you like to see in Nibabel? In-Reply-To: References: <20200728210236.dqmjjl6s5uruvrbm@phare.normalesup.org> <20200729161235.uezevmmxhrd267xk@phare.normalesup.org> Message-ID: <500361B2-0D82-4BED-8B6F-A92CE7EA1DDC@gmail.com> - I second Emanuele?s suggestion. - I would like to add and a community based proposal for a format for tractography it is becoming a necessity. Best regards, Franco FRANCO PESTILLI, PhD | Associate Professor Department of Psychology | College of Liberal Arts | The University of Texas at Austin he/him | pestilli at utexas.edu | web | github | brainlife.io? > On Jul 30, 2020, at 10:59 AM, Emanuele Olivetti wrote: > > Hi Matthew, > > My colleagues and I would like to have a (much) faster loader of tractography data with a smaller memory footprint, especially for TRK (Trackvis) files. Just to give you an idea - which, yes, it's an extreme case but definitely not so infrequent for us - when loading a very large tractogram of 10 million streamlines it takes: > - 15 minutes and approximately 10Gb of RAM with Nibabel > - 1 minute and approximately 5Gb of RAM with our custom TRK loader (it works but code style is meh, https://github.com/FBK-NILab/nilab/blob/master/load_trk.py ) > - 6 seconds and 3.5Gb of RAM if we previously save the tractogram as numpy array (numpy.savez()) and then load it with numpy.load(). But in this case, the streamlines are resampled to a fixed number of points. > > Thank you for all the work done! > > Emanuele > > > On Thu, Jul 30, 2020 at 5:01 PM Matthew Brett > wrote: > Hi, > > Although this is pushing things close the wire, I am currently trying > to work out whether to apply for a Chan Zuckerberg grant for Nibabel: > > https://chanzuckerberg.com/wp-content/uploads/2020/06/EOSS-3-RFA-Announcement-and-Instructions-CYCLE-3-COMBINED-1.pdf > > Before we do that, I wanted to ask - if y'all could get anything into > Nibabel - what > would it be? > > So far, in various conversations, Chris Markiewicz and I came up with: > > * More comprehensive DICOM to NIfTI etc support > * Labelled axes for images, maybe via xarray > * Integrated API for surface formats [1] > * HDF5 serialization for many image formats > * Ability to read images directly from URLs, read / write images to / > from AWS, GCS, Azure data storage addresses [2] > * Optimized visualization for web and desktop > > Is there anything else you'd like to see? > > See you, > > Matthew > > [1] https://github.com/nipy/nibabel/issues/936 > [2] c.f. https://pandas.pydata.org/pandas-docs/version/1.1.0/whatsnew/v1.1.0.html#fsspec-now-used-for-filesystem-handling > _______________________________________________ > Neuroimaging mailing list > Neuroimaging at python.org > https://mail.python.org/mailman/listinfo/neuroimaging > > > -- > Emanuele Olivetti, Ph.D. > Senior Research Scientist at Bruno Kessler Foundation (FBK-ICT) > NeuroInformatics Laboratory (NILab) http://nilab.fbk.eu > Center for Mind and Brain Sciences (CIMeC), University of Trento > Via delle Regole 101 - 38123 Mattarello (Trento), ITALY > olivetti at fbk.eu - +39 0461314179 > emanuele.olivetti at unitn.it - +39 0461282760 > > > > FBK vi invita a leggere il suo Piano di rientro | FBK invites you to read its Premises Reopening Plan . > > -- > Le informazioni contenute nella presente comunicazione sono di natura privata e come tali sono da considerarsi riservate ed indirizzate esclusivamente ai destinatari indicati e per le finalit? strettamente legate al relativo contenuto. Se avete ricevuto questo messaggio per errore, vi preghiamo di eliminarlo e di inviare una comunicazione all?indirizzo e-mail del mittente. > -- > The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you received this in error, please contact the sender and delete the material. > _______________________________________________ > Neuroimaging mailing list > Neuroimaging at python.org > https://mail.python.org/mailman/listinfo/neuroimaging -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Thu Jul 30 12:14:31 2020 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 30 Jul 2020 17:14:31 +0100 Subject: [Neuroimaging] CZI grant - what would you like to see in Nibabel? In-Reply-To: References: <20200728210236.dqmjjl6s5uruvrbm@phare.normalesup.org> <20200729161235.uezevmmxhrd267xk@phare.normalesup.org> Message-ID: Hi, Emanuele - would you mind opening an issue for this? Can anyone volunteer to lead the new streamlines format group? I'm not an expert at all. Cheers, Matthew On Thu, Jul 30, 2020 at 5:03 PM Emanuele Olivetti wrote: > > Hi Matthew, > > My colleagues and I would like to have a (much) faster loader of tractography data with a smaller memory footprint, especially for TRK (Trackvis) files. Just to give you an idea - which, yes, it's an extreme case but definitely not so infrequent for us - when loading a very large tractogram of 10 million streamlines it takes: > - 15 minutes and approximately 10Gb of RAM with Nibabel > - 1 minute and approximately 5Gb of RAM with our custom TRK loader (it works but code style is meh, https://github.com/FBK-NILab/nilab/blob/master/load_trk.py) > - 6 seconds and 3.5Gb of RAM if we previously save the tractogram as numpy array (numpy.savez()) and then load it with numpy.load(). But in this case, the streamlines are resampled to a fixed number of points. > > Thank you for all the work done! > > Emanuele > > > On Thu, Jul 30, 2020 at 5:01 PM Matthew Brett wrote: >> >> Hi, >> >> Although this is pushing things close the wire, I am currently trying >> to work out whether to apply for a Chan Zuckerberg grant for Nibabel: >> >> https://chanzuckerberg.com/wp-content/uploads/2020/06/EOSS-3-RFA-Announcement-and-Instructions-CYCLE-3-COMBINED-1.pdf >> >> Before we do that, I wanted to ask - if y'all could get anything into >> Nibabel - what >> would it be? >> >> So far, in various conversations, Chris Markiewicz and I came up with: >> >> * More comprehensive DICOM to NIfTI etc support >> * Labelled axes for images, maybe via xarray >> * Integrated API for surface formats [1] >> * HDF5 serialization for many image formats >> * Ability to read images directly from URLs, read / write images to / >> from AWS, GCS, Azure data storage addresses [2] >> * Optimized visualization for web and desktop >> >> Is there anything else you'd like to see? >> >> See you, >> >> Matthew >> >> [1] https://github.com/nipy/nibabel/issues/936 >> [2] c.f. https://pandas.pydata.org/pandas-docs/version/1.1.0/whatsnew/v1.1.0.html#fsspec-now-used-for-filesystem-handling >> _______________________________________________ >> Neuroimaging mailing list >> Neuroimaging at python.org >> https://mail.python.org/mailman/listinfo/neuroimaging > > > > -- > Emanuele Olivetti, Ph.D. > Senior Research Scientist at Bruno Kessler Foundation (FBK-ICT) > NeuroInformatics Laboratory (NILab) http://nilab.fbk.eu > Center for Mind and Brain Sciences (CIMeC), University of Trento > Via delle Regole 101 - 38123 Mattarello (Trento), ITALY > olivetti at fbk.eu - +39 0461314179 > emanuele.olivetti at unitn.it - +39 0461282760 > > > > FBK vi invita a leggere il suo Piano di rientro | FBK invites you to read its Premises Reopening Plan. > > -- > Le informazioni contenute nella presente comunicazione sono di natura privata e come tali sono da considerarsi riservate ed indirizzate esclusivamente ai destinatari indicati e per le finalit? strettamente legate al relativo contenuto. Se avete ricevuto questo messaggio per errore, vi preghiamo di eliminarlo e di inviare una comunicazione all?indirizzo e-mail del mittente. > -- > The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you received this in error, please contact the sender and delete the material. > _______________________________________________ > Neuroimaging mailing list > Neuroimaging at python.org > https://mail.python.org/mailman/listinfo/neuroimaging From frakkopesto at gmail.com Thu Jul 30 12:45:47 2020 From: frakkopesto at gmail.com (Franco Pestilli) Date: Thu, 30 Jul 2020 11:45:47 -0500 Subject: [Neuroimaging] CZI grant - what would you like to see in Nibabel? In-Reply-To: References: <20200728210236.dqmjjl6s5uruvrbm@phare.normalesup.org> <20200729161235.uezevmmxhrd267xk@phare.normalesup.org> Message-ID: I started an issue on the format and cc?ed Ariel Rokem and Eleftherios Garyfallidis the leaders of Dipy. It might be important to loop in others. Franco > On Jul 30, 2020, at 11:14 AM, Matthew Brett wrote: > > Hi, > > Emanuele - would you mind opening an issue for this? > > Can anyone volunteer to lead the new streamlines format group? I'm > not an expert at all. > > Cheers, > > Matthew > > On Thu, Jul 30, 2020 at 5:03 PM Emanuele Olivetti wrote: >> >> Hi Matthew, >> >> My colleagues and I would like to have a (much) faster loader of tractography data with a smaller memory footprint, especially for TRK (Trackvis) files. Just to give you an idea - which, yes, it's an extreme case but definitely not so infrequent for us - when loading a very large tractogram of 10 million streamlines it takes: >> - 15 minutes and approximately 10Gb of RAM with Nibabel >> - 1 minute and approximately 5Gb of RAM with our custom TRK loader (it works but code style is meh, https://github.com/FBK-NILab/nilab/blob/master/load_trk.py) >> - 6 seconds and 3.5Gb of RAM if we previously save the tractogram as numpy array (numpy.savez()) and then load it with numpy.load(). But in this case, the streamlines are resampled to a fixed number of points. >> >> Thank you for all the work done! >> >> Emanuele >> >> >> On Thu, Jul 30, 2020 at 5:01 PM Matthew Brett wrote: >>> >>> Hi, >>> >>> Although this is pushing things close the wire, I am currently trying >>> to work out whether to apply for a Chan Zuckerberg grant for Nibabel: >>> >>> https://chanzuckerberg.com/wp-content/uploads/2020/06/EOSS-3-RFA-Announcement-and-Instructions-CYCLE-3-COMBINED-1.pdf >>> >>> Before we do that, I wanted to ask - if y'all could get anything into >>> Nibabel - what >>> would it be? >>> >>> So far, in various conversations, Chris Markiewicz and I came up with: >>> >>> * More comprehensive DICOM to NIfTI etc support >>> * Labelled axes for images, maybe via xarray >>> * Integrated API for surface formats [1] >>> * HDF5 serialization for many image formats >>> * Ability to read images directly from URLs, read / write images to / >>> from AWS, GCS, Azure data storage addresses [2] >>> * Optimized visualization for web and desktop >>> >>> Is there anything else you'd like to see? >>> >>> See you, >>> >>> Matthew >>> >>> [1] https://github.com/nipy/nibabel/issues/936 >>> [2] c.f. https://pandas.pydata.org/pandas-docs/version/1.1.0/whatsnew/v1.1.0.html#fsspec-now-used-for-filesystem-handling >>> _______________________________________________ >>> Neuroimaging mailing list >>> Neuroimaging at python.org >>> https://mail.python.org/mailman/listinfo/neuroimaging >> >> >> >> -- >> Emanuele Olivetti, Ph.D. >> Senior Research Scientist at Bruno Kessler Foundation (FBK-ICT) >> NeuroInformatics Laboratory (NILab) http://nilab.fbk.eu >> Center for Mind and Brain Sciences (CIMeC), University of Trento >> Via delle Regole 101 - 38123 Mattarello (Trento), ITALY >> olivetti at fbk.eu - +39 0461314179 >> emanuele.olivetti at unitn.it - +39 0461282760 >> >> >> >> FBK vi invita a leggere il suo Piano di rientro | FBK invites you to read its Premises Reopening Plan. >> >> -- >> Le informazioni contenute nella presente comunicazione sono di natura privata e come tali sono da considerarsi riservate ed indirizzate esclusivamente ai destinatari indicati e per le finalit? strettamente legate al relativo contenuto. Se avete ricevuto questo messaggio per errore, vi preghiamo di eliminarlo e di inviare una comunicazione all?indirizzo e-mail del mittente. >> -- >> The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you received this in error, please contact the sender and delete the material. >> _______________________________________________ >> Neuroimaging mailing list >> Neuroimaging at python.org >> https://mail.python.org/mailman/listinfo/neuroimaging > _______________________________________________ > Neuroimaging mailing list > Neuroimaging at python.org > https://mail.python.org/mailman/listinfo/neuroimaging From pieper at isomics.com Thu Jul 30 12:50:54 2020 From: pieper at isomics.com (Steve Pieper) Date: Thu, 30 Jul 2020 12:50:54 -0400 Subject: [Neuroimaging] CZI grant - what would you like to see in Nibabel? In-Reply-To: References: <20200728210236.dqmjjl6s5uruvrbm@phare.normalesup.org> <20200729161235.uezevmmxhrd267xk@phare.normalesup.org> Message-ID: Hi Mathew and all - These are all great ideas. +100 for better python imaging tools! Regarding tractography data formats you may be interested in leveraging the new trako format. Preprint and pypi link below. It leverages glTF/draco compression and has nice features to trade off file size for lossiness and also compresses extra data fields. https://arxiv.org/pdf/2004.13630.pdf https://pypi.org/project/trako/ Cheers, Steve On Thu, Jul 30, 2020 at 12:15 PM Matthew Brett wrote: > Hi, > > Emanuele - would you mind opening an issue for this? > > Can anyone volunteer to lead the new streamlines format group? I'm > not an expert at all. > > Cheers, > > Matthew > > On Thu, Jul 30, 2020 at 5:03 PM Emanuele Olivetti wrote: > > > > Hi Matthew, > > > > My colleagues and I would like to have a (much) faster loader of > tractography data with a smaller memory footprint, especially for TRK > (Trackvis) files. Just to give you an idea - which, yes, it's an extreme > case but definitely not so infrequent for us - when loading a very large > tractogram of 10 million streamlines it takes: > > - 15 minutes and approximately 10Gb of RAM with Nibabel > > - 1 minute and approximately 5Gb of RAM with our custom TRK loader (it > works but code style is meh, > https://github.com/FBK-NILab/nilab/blob/master/load_trk.py) > > - 6 seconds and 3.5Gb of RAM if we previously save the tractogram as > numpy array (numpy.savez()) and then load it with numpy.load(). But in this > case, the streamlines are resampled to a fixed number of points. > > > > Thank you for all the work done! > > > > Emanuele > > > > > > On Thu, Jul 30, 2020 at 5:01 PM Matthew Brett > wrote: > >> > >> Hi, > >> > >> Although this is pushing things close the wire, I am currently trying > >> to work out whether to apply for a Chan Zuckerberg grant for Nibabel: > >> > >> > https://chanzuckerberg.com/wp-content/uploads/2020/06/EOSS-3-RFA-Announcement-and-Instructions-CYCLE-3-COMBINED-1.pdf > >> > >> Before we do that, I wanted to ask - if y'all could get anything into > >> Nibabel - what > >> would it be? > >> > >> So far, in various conversations, Chris Markiewicz and I came up with: > >> > >> * More comprehensive DICOM to NIfTI etc support > >> * Labelled axes for images, maybe via xarray > >> * Integrated API for surface formats [1] > >> * HDF5 serialization for many image formats > >> * Ability to read images directly from URLs, read / write images to / > >> from AWS, GCS, Azure data storage addresses [2] > >> * Optimized visualization for web and desktop > >> > >> Is there anything else you'd like to see? > >> > >> See you, > >> > >> Matthew > >> > >> [1] https://github.com/nipy/nibabel/issues/936 > >> [2] c.f. > https://pandas.pydata.org/pandas-docs/version/1.1.0/whatsnew/v1.1.0.html#fsspec-now-used-for-filesystem-handling > >> _______________________________________________ > >> Neuroimaging mailing list > >> Neuroimaging at python.org > >> https://mail.python.org/mailman/listinfo/neuroimaging > > > > > > > > -- > > Emanuele Olivetti, Ph.D. > > Senior Research Scientist at Bruno Kessler Foundation (FBK-ICT) > > NeuroInformatics Laboratory (NILab) http://nilab.fbk.eu > > Center for Mind and Brain Sciences (CIMeC), University of Trento > > Via delle Regole 101 - 38123 Mattarello (Trento), ITALY > > olivetti at fbk.eu - +39 0461314179 > > emanuele.olivetti at unitn.it - +39 0461282760 > > > > > > > > FBK vi invita a leggere il suo Piano di rientro | FBK invites you to > read its Premises Reopening Plan. > > > > -- > > Le informazioni contenute nella presente comunicazione sono di natura > privata e come tali sono da considerarsi riservate ed indirizzate > esclusivamente ai destinatari indicati e per le finalit? strettamente > legate al relativo contenuto. Se avete ricevuto questo messaggio per > errore, vi preghiamo di eliminarlo e di inviare una comunicazione > all?indirizzo e-mail del mittente. > > -- > > The information transmitted is intended only for the person or entity to > which it is addressed and may contain confidential and/or privileged > material. If you received this in error, please contact the sender and > delete the material. > > _______________________________________________ > > Neuroimaging mailing list > > Neuroimaging at python.org > > https://mail.python.org/mailman/listinfo/neuroimaging > _______________________________________________ > Neuroimaging mailing list > Neuroimaging at python.org > https://mail.python.org/mailman/listinfo/neuroimaging > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Reid.Robert at mayo.edu Thu Jul 30 12:50:00 2020 From: Reid.Robert at mayo.edu (Reid, Robert I. (Rob)) Date: Thu, 30 Jul 2020 16:50:00 +0000 Subject: [Neuroimaging] [EXTERNAL] Re: CZI grant - what would you like to see in Nibabel? In-Reply-To: References: <20200728210236.dqmjjl6s5uruvrbm@phare.normalesup.org> <20200729161235.uezevmmxhrd267xk@phare.normalesup.org> Message-ID: <28fddd$e7po4d@ironport10.mayo.edu> Hi, I third the tractogram request, at the risk of taking the thread off topic. Would it be in nibabel?s remit to add a function to view a tractogram as a 3D array of streamline counts per voxel? That way NIFTI viewers such as FSLeyes that are not (last I checked) set up as tractogram viewers a la trackvis, but are python powered, could load and display tractograms in a limited way on the fly. I sometimes find it handy to overlay tractogram with other images using a viewer that caters more to the traditional ?2D? display mode with lots of overlay options than one that focuses on 3D renders. I got a 404 error for Emanuele?s load_trk.py URL. Maybe things were rearranged? Could near numpy performance without resampling be achieved by storing the tractogram as a dict of numpy arrays keyed by the number of points in their streamlines? e.g. {20: , 21: , ?} Thanks, Rob From: Neuroimaging [mailto:neuroimaging-bounces+reid.robert=mayo.edu at python.org] On Behalf Of Emanuele Olivetti Sent: Thursday, July 30, 2020 11:00 AM To: Neuroimaging analysis in Python Subject: [EXTERNAL] Re: [Neuroimaging] CZI grant - what would you like to see in Nibabel? Hi Matthew, My colleagues and I would like to have a (much) faster loader of tractography data with a smaller memory footprint, especially for TRK (Trackvis) files. Just to give you an idea - which, yes, it's an extreme case but definitely not so infrequent for us - when loading a very large tractogram of 10 million streamlines it takes: - 15 minutes and approximately 10Gb of RAM with Nibabel - 1 minute and approximately 5Gb of RAM with our custom TRK loader (it works but code style is meh, https://github.com/FBK-NILab/nilab/blob/master/load_trk.py) - 6 seconds and 3.5Gb of RAM if we previously save the tractogram as numpy array (numpy.savez()) and then load it with numpy.load(). But in this case, the streamlines are resampled to a fixed number of points. Thank you for all the work done! Emanuele On Thu, Jul 30, 2020 at 5:01 PM Matthew Brett > wrote: Hi, Although this is pushing things close the wire, I am currently trying to work out whether to apply for a Chan Zuckerberg grant for Nibabel: https://chanzuckerberg.com/wp-content/uploads/2020/06/EOSS-3-RFA-Announcement-and-Instructions-CYCLE-3-COMBINED-1.pdf Before we do that, I wanted to ask - if y'all could get anything into Nibabel - what would it be? So far, in various conversations, Chris Markiewicz and I came up with: * More comprehensive DICOM to NIfTI etc support * Labelled axes for images, maybe via xarray * Integrated API for surface formats [1] * HDF5 serialization for many image formats * Ability to read images directly from URLs, read / write images to / from AWS, GCS, Azure data storage addresses [2] * Optimized visualization for web and desktop Is there anything else you'd like to see? See you, Matthew [1] https://github.com/nipy/nibabel/issues/936 [2] c.f. https://pandas.pydata.org/pandas-docs/version/1.1.0/whatsnew/v1.1.0.html#fsspec-now-used-for-filesystem-handling _______________________________________________ Neuroimaging mailing list Neuroimaging at python.org https://mail.python.org/mailman/listinfo/neuroimaging -- Emanuele Olivetti, Ph.D. Senior Research Scientist at Bruno Kessler Foundation (FBK-ICT) NeuroInformatics Laboratory (NILab) http://nilab.fbk.eu Center for Mind and Brain Sciences (CIMeC), University of Trento Via delle Regole 101 - 38123 Mattarello (Trento), ITALY olivetti at fbk.eu - +39 0461314179 emanuele.olivetti at unitn.it - +39 0461282760 FBK vi invita a leggere il suo Piano di rientro | FBK invites you to read its Premises Reopening Plan. -- Le informazioni contenute nella presente comunicazione sono di natura privata e come tali sono da considerarsi riservate ed indirizzate esclusivamente ai destinatari indicati e per le finalit? strettamente legate al relativo contenuto. Se avete ricevuto questo messaggio per errore, vi preghiamo di eliminarlo e di inviare una comunicazione all?indirizzo e-mail del mittente. -- The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you received this in error, please contact the sender and delete the material. -------------- next part -------------- An HTML attachment was scrubbed... URL: From olivetti at fbk.eu Thu Jul 30 16:56:21 2020 From: olivetti at fbk.eu (Emanuele Olivetti) Date: Thu, 30 Jul 2020 22:56:21 +0200 Subject: [Neuroimaging] [EXTERNAL] Re: CZI grant - what would you like to see in Nibabel? In-Reply-To: <28fddd$e7po4d@ironport10.mayo.edu> References: <20200728210236.dqmjjl6s5uruvrbm@phare.normalesup.org> <20200729161235.uezevmmxhrd267xk@phare.normalesup.org> <28fddd$e7po4d@ironport10.mayo.edu> Message-ID: On Thu, Jul 30, 2020 at 6:55 PM Reid, Robert I. (Rob) via Neuroimaging < neuroimaging at python.org> wrote: > [...] > > > > I got a 404 error for Emanuele?s load_trk.py URL. Maybe things were > rearranged? > My mistake, that was a private repo with some code in progress. I've made available the relevant (and self-contained) file from here: https://github.com/emanuele/load_trk.git > > > Could near numpy performance without resampling be achieved by storing the > tractogram as a dict of numpy arrays keyed by the number of points in their > streamlines? > > e.g. {20: , 21: array of all the streamlines with 21 points>, ?} > > > Good question and interesting suggestion. Nevertheless, in most of our work, we need an easy way to access the coordinates of streamlines and to have a unique ID for each streamline. In years of experiments and coding, we almost always ended up with one of these two simple data structures: 1) In the case of a tractogram (T) where the streamlines have mixed number of points: a numpy.array of M elements (like M=10 millions) and dtype=np.object, where each element/object is a streamline, i.e. a matrix n x 3, where n is its number of points, that may change from streamline to streamline. 2) In the case of tractogram (T) where the streamlines all have the same number of points, e.g. 16: a numpy array M x 16 x 3, typically dtype=numpy.float32 to save some space (remember the 10M streamlines? It's 2Gb). Why numpy.array? Mainly for the very convenient indexing property. First, each streamline has a unique ID (in the example above an int between 0 and 9999999), which is the position of the streamline in the array. So if, for example, we write an algorithm for nearest neighbour and compute the 100 nearest neighbours to the streamline having ID=123456, we just need a list or numpy.array (neighbours) of 100 integers to store the result. If we need to retrieve those neighbouring streamlines, it's just "T[neighbours]", which is also very fast. Moreover, many algorithms in data analysis / machine learning operate directly on numpy.arrays. A side note: At the repo above, I've just added a small test. Loading 4 million [*] streamlines with mixed number of points takes 1 minute with our load_streamlines() and 6 seconds with numpy.load() - not fat from what I reported in the previous message, where instead all streamlines had the same number of points. Best, Emanuele [*]:In this case, the problem is the excessive use of RAM when doing numpy.save() - that's why I could test just 4 millions on my laptop. -- _FBK vi invita a leggere il suo?Piano di rientro ? | FBK invites you to read its?Premises Reopening Plan ._ -- Le informazioni contenute nella presente comunicazione sono di natura? privata e come tali sono da considerarsi riservate ed indirizzate? esclusivamente ai destinatari indicati e per le finalit? strettamente? legate al relativo contenuto. Se avete ricevuto questo messaggio per? errore, vi preghiamo di eliminarlo e di inviare una comunicazione? all?indirizzo e-mail del mittente. -- The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you received this in error, please contact the sender and delete the material. -------------- next part -------------- An HTML attachment was scrubbed... URL: From olivetti at fbk.eu Thu Jul 30 16:56:55 2020 From: olivetti at fbk.eu (Emanuele Olivetti) Date: Thu, 30 Jul 2020 22:56:55 +0200 Subject: [Neuroimaging] CZI grant - what would you like to see in Nibabel? In-Reply-To: References: <20200728210236.dqmjjl6s5uruvrbm@phare.normalesup.org> <20200729161235.uezevmmxhrd267xk@phare.normalesup.org> Message-ID: Will do. Most probably tomorrow. Cheers, Emanuele On Thu, Jul 30, 2020 at 6:15 PM Matthew Brett wrote: > Hi, > > Emanuele - would you mind opening an issue for this? > > Can anyone volunteer to lead the new streamlines format group? I'm > not an expert at all. > > Cheers, > > Matthew > > On Thu, Jul 30, 2020 at 5:03 PM Emanuele Olivetti wrote: > > > > Hi Matthew, > > > > My colleagues and I would like to have a (much) faster loader of > tractography data with a smaller memory footprint, especially for TRK > (Trackvis) files. Just to give you an idea - which, yes, it's an extreme > case but definitely not so infrequent for us - when loading a very large > tractogram of 10 million streamlines it takes: > > - 15 minutes and approximately 10Gb of RAM with Nibabel > > - 1 minute and approximately 5Gb of RAM with our custom TRK loader (it > works but code style is meh, > https://github.com/FBK-NILab/nilab/blob/master/load_trk.py) > > - 6 seconds and 3.5Gb of RAM if we previously save the tractogram as > numpy array (numpy.savez()) and then load it with numpy.load(). But in this > case, the streamlines are resampled to a fixed number of points. > > > > Thank you for all the work done! > > > > Emanuele > > > > > > On Thu, Jul 30, 2020 at 5:01 PM Matthew Brett > wrote: > >> > >> Hi, > >> > >> Although this is pushing things close the wire, I am currently trying > >> to work out whether to apply for a Chan Zuckerberg grant for Nibabel: > >> > >> > https://chanzuckerberg.com/wp-content/uploads/2020/06/EOSS-3-RFA-Announcement-and-Instructions-CYCLE-3-COMBINED-1.pdf > >> > >> Before we do that, I wanted to ask - if y'all could get anything into > >> Nibabel - what > >> would it be? > >> > >> So far, in various conversations, Chris Markiewicz and I came up with: > >> > >> * More comprehensive DICOM to NIfTI etc support > >> * Labelled axes for images, maybe via xarray > >> * Integrated API for surface formats [1] > >> * HDF5 serialization for many image formats > >> * Ability to read images directly from URLs, read / write images to / > >> from AWS, GCS, Azure data storage addresses [2] > >> * Optimized visualization for web and desktop > >> > >> Is there anything else you'd like to see? > >> > >> See you, > >> > >> Matthew > >> > >> [1] https://github.com/nipy/nibabel/issues/936 > >> [2] c.f. > https://pandas.pydata.org/pandas-docs/version/1.1.0/whatsnew/v1.1.0.html#fsspec-now-used-for-filesystem-handling > >> _______________________________________________ > >> Neuroimaging mailing list > >> Neuroimaging at python.org > >> https://mail.python.org/mailman/listinfo/neuroimaging > > > > > > > > -- > > Emanuele Olivetti, Ph.D. > > Senior Research Scientist at Bruno Kessler Foundation (FBK-ICT) > > NeuroInformatics Laboratory (NILab) http://nilab.fbk.eu > > Center for Mind and Brain Sciences (CIMeC), University of Trento > > Via delle Regole 101 - 38123 Mattarello (Trento), ITALY > > olivetti at fbk.eu - +39 0461314179 > > emanuele.olivetti at unitn.it - +39 0461282760 > > > > > > > > FBK vi invita a leggere il suo Piano di rientro | FBK invites you to > read its Premises Reopening Plan. > > > > -- > > Le informazioni contenute nella presente comunicazione sono di natura > privata e come tali sono da considerarsi riservate ed indirizzate > esclusivamente ai destinatari indicati e per le finalit? strettamente > legate al relativo contenuto. Se avete ricevuto questo messaggio per > errore, vi preghiamo di eliminarlo e di inviare una comunicazione > all?indirizzo e-mail del mittente. > > -- > > The information transmitted is intended only for the person or entity to > which it is addressed and may contain confidential and/or privileged > material. If you received this in error, please contact the sender and > delete the material. > > _______________________________________________ > > Neuroimaging mailing list > > Neuroimaging at python.org > > https://mail.python.org/mailman/listinfo/neuroimaging > _______________________________________________ > Neuroimaging mailing list > Neuroimaging at python.org > https://mail.python.org/mailman/listinfo/neuroimaging > -- _FBK vi invita a leggere il suo?Piano di rientro ? | FBK invites you to read its?Premises Reopening Plan ._ -- Le informazioni contenute nella presente comunicazione sono di natura? privata e come tali sono da considerarsi riservate ed indirizzate? esclusivamente ai destinatari indicati e per le finalit? strettamente? legate al relativo contenuto. Se avete ricevuto questo messaggio per? errore, vi preghiamo di eliminarlo e di inviare una comunicazione? all?indirizzo e-mail del mittente. -- The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you received this in error, please contact the sender and delete the material. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Reid.Robert at mayo.edu Thu Jul 30 17:32:44 2020 From: Reid.Robert at mayo.edu (Reid, Robert I. (Rob)) Date: Thu, 30 Jul 2020 21:32:44 +0000 Subject: [Neuroimaging] [EXTERNAL] Re: Re: CZI grant - what would you like to see in Nibabel? In-Reply-To: References: <20200728210236.dqmjjl6s5uruvrbm@phare.normalesup.org> <20200729161235.uezevmmxhrd267xk@phare.normalesup.org> <28fddd$e7po4d@ironport10.mayo.edu> Message-ID: <28fddd$e7sndt@ironport10.mayo.edu> Thanks! From: Neuroimaging [mailto:neuroimaging-bounces+reid.robert=mayo.edu at python.org] On Behalf Of Emanuele Olivetti Sent: Thursday, July 30, 2020 3:56 PM To: Neuroimaging analysis in Python Subject: [EXTERNAL] Re: [Neuroimaging] Re: CZI grant - what would you like to see in Nibabel? On Thu, Jul 30, 2020 at 6:55 PM Reid, Robert I. (Rob) via Neuroimaging > wrote: [...] I got a 404 error for Emanuele?s load_trk.py URL. Maybe things were rearranged? My mistake, that was a private repo with some code in progress. I've made available the relevant (and self-contained) file from here: https://github.com/emanuele/load_trk.git Could near numpy performance without resampling be achieved by storing the tractogram as a dict of numpy arrays keyed by the number of points in their streamlines? e.g. {20: , 21: , ?} Good question and interesting suggestion. Nevertheless, in most of our work, we need an easy way to access the coordinates of streamlines and to have a unique ID for each streamline. In years of experiments and coding, we almost always ended up with one of these two simple data structures: 1) In the case of a tractogram (T) where the streamlines have mixed number of points: a numpy.array of M elements (like M=10 millions) and dtype=np.object, where each element/object is a streamline, i.e. a matrix n x 3, where n is its number of points, that may change from streamline to streamline. 2) In the case of tractogram (T) where the streamlines all have the same number of points, e.g. 16: a numpy array M x 16 x 3, typically dtype=numpy.float32 to save some space (remember the 10M streamlines? It's 2Gb). Why numpy.array? Mainly for the very convenient indexing property. First, each streamline has a unique ID (in the example above an int between 0 and 9999999), which is the position of the streamline in the array. So if, for example, we write an algorithm for nearest neighbour and compute the 100 nearest neighbours to the streamline having ID=123456, we just need a list or numpy.array (neighbours) of 100 integers to store the result. If we need to retrieve those neighbouring streamlines, it's just "T[neighbours]", which is also very fast. Moreover, many algorithms in data analysis / machine learning operate directly on numpy.arrays. A side note: At the repo above, I've just added a small test. Loading 4 million [*] streamlines with mixed number of points takes 1 minute with our load_streamlines() and 6 seconds with numpy.load() - not fat from what I reported in the previous message, where instead all streamlines had the same number of points. Best, Emanuele [*]:In this case, the problem is the excessive use of RAM when doing numpy.save() - that's why I could test just 4 millions on my laptop. FBK vi invita a leggere il suo Piano di rientro | FBK invites you to read its Premises Reopening Plan. -- Le informazioni contenute nella presente comunicazione sono di natura privata e come tali sono da considerarsi riservate ed indirizzate esclusivamente ai destinatari indicati e per le finalit? strettamente legate al relativo contenuto. Se avete ricevuto questo messaggio per errore, vi preghiamo di eliminarlo e di inviare una comunicazione all?indirizzo e-mail del mittente. -- The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you received this in error, please contact the sender and delete the material. -------------- next part -------------- An HTML attachment was scrubbed... URL: