[AstroPy] ValueError in vstacking tables
Thomas Robitaille
thomas.robitaille at gmail.com
Tue Jun 23 10:10:22 EDT 2015
Hi Brian,
Thanks, this is helpful, and I managed to boil this down to a simple
failing example that I've reported here:
https://github.com/astropy/astropy/issues/3877
It seems to be due to the vector columns. We'll look into it!
Cheers,
Tom
On 23 June 2015 at 14:58, Brian Cherinka <cherinka at di.utoronto.ca> wrote:
> Hi Thomas,
>
> In [187]: t1.dtype
> Out[187]: dtype([('plate', '>i8'), ('ifudsgn', 'S5'), ('plateifu', 'S9'),
> ('mangaid', 'S4'), ('versdrp2', 'S30'), ('versdrp3', 'S30'), ('verscore',
> 'S30'), ('versutil', 'S30'), ('platetyp', 'S30'), ('srvymode', 'S30'),
> ('objRA', '>f8'), ('objDEC', '>f8'), ('objGLON', '>f8'), ('objGLAT', '>f8'),
> ('ebvgal', '>f8'), ('nexp', '>i8'), ('exptime', '>f8'), ('drp3qual', '>i8'),
> ('bluesn2', '>f8'), ('redsn2', '>f8'), ('harname', 'S30'), ('frlplug',
> '>i8'), ('cartid', 'S30'), ('designid', '>i8'), ('cenRA', '>f8'), ('cenDEC',
> '>f8'), ('airmsmin', '>f8'), ('airmsmed', '>f8'), ('airmsmax', '>f8'),
> ('seemin', '>f8'), ('seemed', '>f8'), ('seemax', '>f8'), ('transmin',
> '>f8'), ('transmed', '>f8'), ('transmax', '>f8'), ('mjdmin', '>i8'),
> ('mjdmed', '>i8'), ('mjdmax', '>i8'), ('ufwhm', '>f8'), ('gfwhm', '>f8'),
> ('rfwhm', '>f8'), ('ifwhm', '>f8'), ('zfwhm', '>f8'), ('mngtarg1', '>i8'),
> ('mngtarg2', '>i8'), ('mngtarg3', '>i8'), ('catidnum', '>i8'), ('plttarg',
> 'S30')])
>
> In [188]: t2.dtype
> Out[188]: dtype([('manga_tileid', '<i8'), ('ifu_ra', '<f8'), ('ifu_dec',
> '<f8'), ('target_ra', '<f8'), ('target_dec', '<f8'), ('iauname', 'S19'),
> ('ifudesignsize', '<i8'), ('ifutargetsize', '<i8'), ('ifudesignwrongsize',
> '<i8'), ('field', '<i8'), ('run', '<i8'), ('nsa_version', 'S6'), ('nsa_id',
> '<i8'), ('nsa_id100', '<i8'), ('nsa_redshift', '<f8'), ('nsa_zdist', '<f8'),
> ('nsa_absmag', '<f8', (7,)), ('nsa_absmag_el', '<f8', (7,)),
> ('nsa_amivar_el', '<f8', (7,)), ('nsa_mstar', '<f8'), ('nsa_mstar_el',
> '<f8'), ('nsa_ba', '<f8'), ('nsa_phi', '<f8'), ('nsa_extinction', '<f8',
> (7,)), ('nsa_petro_th50', '<f8'), ('nsa_petro_th50_el', '<f8'),
> ('nsa_petroflux', '<f8', (7,)), ('nsa_petroflux_ivar', '<f8', (7,)),
> ('nsa_petroflux_el', '<f8', (7,)), ('nsa_petroflux_el_ivar', '<f8', (7,)),
> ('nsa_sersic_ba', '<f8'), ('nsa_sersic_n', '<f8'), ('nsa_sersic_phi',
> '<f8'), ('nsa_sersic_th50', '<f8'), ('nsa_sersicflux', '<f8', (7,)),
> ('nsa_sersicflux_ivar', '<f8', (7,))])
>
> t3 = hstack([t1,t2],join_type='outer’) completes with no errors, however
>
> t3[0] gives
>
> <repr(<astropy.table.row.Row at 0x108af0b90>) failed: ValueError: setting an
> array element with a sequence.>
>
> then vstacking t3 with t4 completes with no errors, but gives the same error
> when row indexing.
>
> Cheers, Brian
>
> ---------------------------------------
> Brian Cherinka, Ph.D
> Dunlap Institute for Astronomy & Astrophysics
> University of Toronto
> Toronto, ON, Canada M5S 3H4
> phone: 416-978-7299
> email: cherinka at di.utoronto.ca
> ----------------------------------------
>
> On Jun 23, 2015, at 9:53 AM, Thomas Robitaille <thomas.robitaille at gmail.com>
> wrote:
>
> Hi Brian,
>
> On 23 June 2015 at 14:49, Brian Cherinka <cherinka at di.utoronto.ca> wrote:
>
> Hi Thomas,
>
> Could you let us know what table1.dtype and table3.dtype are equal to,
> and how many rows are in each table?
>
>
> The two tables, t1 and t2, that I am hstacking have 48 columns, and 36
> columns, respectively, each with 29 rows. The dtype of those columns are a
> mix of things. Mostly float and string scalars. I do have several columns
> of float arrays.
>
>
> Just to clarify, could you copy and paste the exact output from
> table1.dtype and table3.dtype here so that we can try and reproduce
> the exact issue? (it may be a corner case due to the type of one of
> the columns).
>
> Thanks!
> Tom
>
> After hstacking, the resulting table (84 columns, 29 rows) is combined as
> expected, although I can no longer index the rows. Then I vstack it with
> another table 3, which has the same number of columns, and same dtype. I
> can’t row index that table either.
>
>
>
>
> Cheers, Brian
>
>
> ---------------------------------------
> Brian Cherinka, Ph.D
> Dunlap Institute for Astronomy & Astrophysics
> University of Toronto
> Toronto, ON, Canada M5S 3H4
> phone: 416-978-7299
> email: cherinka at di.utoronto.ca
> ----------------------------------------
>
> On Jun 19, 2015, at 8:43 AM, Thomas Robitaille <thomas.robitaille at gmail.com>
> wrote:
>
> Hi Brian,
>
> Could you let us know what table1.dtype and table3.dtype are equal to,
> and how many rows are in each table?
>
> Thanks!
> Tom
>
>
> On 18 June 2015 at 16:34, Brian Cherinka <cherinka at di.utoronto.ca> wrote:
>
> Anymore thoughts on this? My table length issue wasn’t the problem. I’m
> stilling having a problem with my resulting hstacks and vstacks. The stacks
> happen without error. I just lose the ability to index by row.
>
> Cheers, Brian
>
> ---------------------------------------
> Brian Cherinka, Ph.D
> Dunlap Institute for Astronomy & Astrophysics
> University of Toronto
> Toronto, ON, Canada M5S 3H4
> phone: 416-978-7299
> email: cherinka at di.utoronto.ca
> ----------------------------------------
>
> On Jun 17, 2015, at 3:25 PM, Brian Cherinka <cherinka at di.utoronto.ca> wrote:
>
> Hi Thomas,
>
> Your example does indeed work for me. I cannot yet guess why it doesn’t
> work in my full example. However it’s not quite a complete reconstruction
> of what I’m doing. Technically, I’m hstacking two tables together, and then
> vstacking that result with another table. Something like this, which works
> just fine.
>
> In [382]: t1 = Table([[1], [2]], names=['a', 'b'])
>
> In [383]: t2 = Table([[3], [4]], names=['c','d'])
>
> In [384]: t3 = hstack([t1,t2],join_type='outer')
>
> In [386]: t4 = Table([[5], [6]], names=['a', 'b'])
>
> In [387]: t34 = vstack([t3,t4],join_type='outer')
>
> In [388]: t34
> Out[388]:
> <Table masked=True length=2>
> a b c d
> int64 int64 int64 int64
> ----- ----- ----- -----
> 1 2 3 4
> 5 6 -- --
>
> In [389]: t34[0]
> Out[389]:
> <Row 0 of table
> values=(1, 2, 3, 4)
> dtype=[('a', '<i8'), ('b', '<i8'), ('c', '<i8'), ('d', '<i8')]>
>
> In my full example, I did find though that I can row index the individual
> tables t1 and t2, but cannot row index table t3 after hstacking. I get the
> same ValueError. When I attempt to vstack with table t4, that error
> propagates forward. I can row index t4 but not t34. Looking at it a bit
> more, it might be that my rows in the initial tables t1, t2 do not match up
> with one another, and I appear to have one extra row in table t2, than I
> should. I’ll dig a bit more.
>
> Cheers, Brian
>
> ---------------------------------------
> Brian Cherinka, Ph.D
> Dunlap Institute for Astronomy & Astrophysics
> University of Toronto
> Toronto, ON, Canada M5S 3H4
> phone: 416-978-7299
> email: cherinka at di.utoronto.ca
> ----------------------------------------
>
> On Jun 17, 2015, at 2:20 PM, Aldcroft, Thomas
> <aldcroft at head.cfa.harvard.edu> wrote:
>
> Hi Brian,
>
> Below is what I think is a minimal example of what you are describing. Can
> you look at this and first reproduce that this works for you, and second,
> can you guess what might be different in your full example vs. my minimal
> example?
>
> In [19]: t1 = Table([[1], [2]], names=['a', 'b'])
>
> In [20]: t2 = Table([[3], [4]], names=['b', 'c'])
>
> In [23]: t12 = vstack([t1, t2], join_type='outer')
>
> In [24]: t12
> Out[24]:
> <Table masked=True length=2>
> a b c
> int64 int64 int64
> ----- ----- -----
> 1 2 --
> -- 3 4
>
> In [25]: t12[0]
> Out[25]:
> <Row 0 of table
> values=(1, 2, --)
> dtype=[('a', '<i8'), ('b', '<i8'), ('c', '<i8')]>
>
>
> On Wed, Jun 17, 2015 at 2:08 PM, Brian Cherinka <cherinka at di.utoronto.ca>
> wrote:
>
>
> Hi,
>
> I’m trying to vstack (vertically stack) two Astropy tables with different
> numbers of columns. The tables appear to stack correctly without error.
> However, upon attempting to index a row in the table, it throws me this
> error
>
> <repr(<astropy.table.row.Row at 0x10c72ed50>) failed: ValueError: setting
> an array element with a sequence.>
>
> I’m not trying to do anything fancy with mixed columns, or Quantities.
> Just stackin’ two good ol’ natural Tables together. One table has 84
> columns, and another has 57, which are all a part of the 84 in the first
> table. At first glance I would guess it’s a mismatch in the dtypes of a
> particular column that I’m trying to merge. However, the columns all have
> the same dtype, format, and they are in the same order. I get no messages
> regarding any kind of merge errors.
>
> Any ideas on this?
>
> Thanks, Brian
>
> P.S. Full print of traceback below
>
> table3 = vstack([table1, table3],join_type=‘outer’)
> print table3[0]
>
> ---------------------------------------------------------------------------
> ValueError Traceback (most recent call
> last)
> <ipython-input-278-a0c75e8d3ef3> in <module>()
> ----> 1 print tmp3[0]
>
> /Users/Brian/anaconda/lib/python2.7/site-packages/astropy/table/row.pyc in
> __repr__(self)
> 176 def __repr__(self):
> 177 return "<{3} {0} of table\n values={1!r}\n
> dtype={2}>".format(
> --> 178 self.index, self.as_void(), self.dtype,
> self.__class__.__name__)
> 179
> 180
>
> /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in
> __repr__(self)
> 5699 return self._data.__repr__()
> 5700 m = tuple(m)
> -> 5701 if not any(m):
> 5702 return self._data.__repr__()
> 5703 p = masked_print_option
>
> /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in
> __call__(self, a, *args, **params)
> 6089 method = getattr(MaskedArray, method_name, None)
> 6090 if method is not None:
> -> 6091 return method(MaskedArray(a), *args, **params)
> 6092 # Still here ? OK, let's call the corresponding np
> function
> 6093 method = getattr(np, method_name)
>
> /Users/Brian/anaconda/lib/python2.7/site-packages/numpy/ma/core.pyc in
> __new__(cls, data, mask, dtype, copy, subok, ndmin, fill_value, keep_mask,
> hard_mask, shrink, **options)
> 2649 """
> 2650 # Process data............
> -> 2651 _data = np.array(data, dtype=dtype, copy=copy, subok=True,
> ndmin=ndmin)
> 2652 _baseclass = getattr(data, '_baseclass', type(_data))
> 2653 # Check that we're not erasing the mask..........
>
> ValueError: setting an array element with a sequence.
>
>
>
>
> ---------------------------------------
> Brian Cherinka, Ph.D
> Dunlap Institute for Astronomy & Astrophysics
> University of Toronto
> Toronto, ON, Canada M5S 3H4
> phone: 416-978-7299
> email: cherinka at di.utoronto.ca
> ----------------------------------------
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
>
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
>
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
>
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at scipy.org
> http://mail.scipy.org/mailman/listinfo/astropy
>
More information about the AstroPy
mailing list