[Pandas-dev] A case for a simplified (non-consolidating) BlockManager with 1D blocks

[Joris Van den Bossche]

On Wed, 27 May 2020 at 23:07, Brock Mendel <jbrockmendel at gmail.com> wrote:

> > I don't think this "lazy _mgr attribute" is comparable in complexity
> with the consolidated BlockManager
>
> Not on its own, no.  But my prior is that this isn't the last thing that
> will merit its own special case.
>
> > I think it clear that a BlockManager with only 1D arrays/blocks *can* be
> simpler as one with interleaved/consolidated blocks.
>
> Absolutely agree.  I've spent a big chunk of the last year dealing with
> BlockManager code and have no great love for it.
>
> > But this is also only one of the arguments. Complexity alone is not a
> reason to not do something; it's the general trade-off with what you gain
> or lose with it.
>
> The main upsides I see are a) internal complexity reduction, b) downstream
> library upsides, c) clearer view vs copy semantics, d) perf improvements
> from making fewer copies, e) clear "dict of Series" data model.
>
> The main downside is potential performance degradation (at the extreme end
> e.g. 3000x <https://github.com/pandas-dev/pandas/issues/24990> for
> arithmetic).  As Wes commented some of that can be ameliorated with
> compiled code but that cuts against the complexity reduction.
>

That number is not correct. That was comparing the block-wise operation to
a very inefficient convert-each-column-to-a-series operation. We can
optimize this column-wise operation a lot (as I already did on master for
some cases), and then a slowdown will still be present in such extreme
cases, but *much* less.


>
> I am looking for ways to quantify these tradeoffs so we can make an
> informed decision.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pandas-dev/attachments/20200527/0d21c55f/attachment-0001.html>