[C++-sig] Re: New slice implementation

[Jonathan Brandmeyer]

On Thu, 2004-01-08 at 20:55, Raoul Gough wrote:
> Jonathan Brandmeyer <jbrandmeyer at earthlink.net> writes:
> 
> > On Thu, 2004-01-08 at 07:22, Raoul Gough wrote:
> > >
> > double
> > partial_sum( std::vector<double>* Foo, slice index)
> > {
> >     slice::range<std::vector<double> > bounds;
> >     try {
> >         bounds = index.get_indicies( Foo->begin(), Foo->end());
> >     } catch (std::invalid_argument)
> >         return 0.0;
> >     double ret = 0.0;
> >     while (bounds.start != bounds.stop) {
> >         ret += *bounds.start;
> >         std::advance( bounds.start, bounds.step);
> >     }
> >     ret += bounds.start;
> >     return ret;
> > }
> 
> Yes, I guess it makes a good deal of sense to use iterators in this
> case.  However, how would you make use of them from Python code?

The idea is that you don't make use of the iterators from Python (Python
has no notion of an iterator pair, anyway), you use the slice object to
define the portion of the container to act upon as a kind of replacement
for iterator pairs.

Based on this question and the one about raw PySliceObject*'s, I don't
think I've explained myself enough.  Here is the rest of a complete,
working example that uses the existing vector_indexing_suite and the
patches I've proposed.

// Add in the code from the above example, plus slice.hpp, slice.cpp, 
// and the patches I sent for slice_nil.hpp and object_core.hpp
// Also add in the injected constructor that I wrote below in the
// last message
#include <boost/python.hpp>
#include <boost/python/slice.hpp>
#include <boost/python/suite/indexing/vector_indexing_suite.hpp>
#include <boost/python/make_constructor.hpp>

BOOST_PYTHON_MODULE(vector_test)
{
    using namespace boost::python;

    class_<std::vector<double> >( "float_vector")
        .def( "__init__", make_constructor(
             create_from_pysequence<std::vector<double> >))
        .def( vector_indexing_suite<std::vector<double> >())
        .def( "partial_sum", &partial_sum)
        ;
}

----end file----
And now, in Python you can do this:
>>> x = float_vector( [1,2,3,4,5,6])
>>> x.partial_sum( slice(None,None))
21
>>> x.partial_sum( slice(3,-1))
9
>>> x[3:].partial_sum(slice(None,None))
15


> >> BTW, wouldn't it be a good idea to have a slice constructor that takes
> >> a PySliceObject * as parameter?
> >
> > I try to avoid raw PyObject*'s whenever possible, but I think that the
> > answer is "no".  The reason is that you have no idea how to properly
> > manage it.  That is partially what the detail::new_reference<>,
> > detail::borrowed_reference<>, and detail::new_non_null_reference<> are
> > for, right?.  Feel free to correct me if I'm wrong.
> 
> I didn't necessarily mean a raw PySliceObject. All I'm getting at is,
> if you want to implement __getitem__ then you will end up with a
> PySliceObject created by the Python interpreter. AFAICS, you don't
> have a way of generating one of your slice objects from this. Am I
> missing something here?
> 
> >>> part = v[1::4]
> 
> calls v.__getitem__ with a PySliceObject (1, None, 4)

*I* won't end up with a PySliceObject created by the Python interpreter,
I will end up with a boost::python::slice object manager that has been
automatically converted by def() and/or class_::def() from the original
PySliceObject.

> >> > ---crash_test.py---
> >> > # Uses the existing vector_indexing_suite_ext.cpp test modul
> >> > from vector_indexing_suite_ext import *
> >> > foo = FloatVec()
> >> > # Weird initialization, why not supported by a constructor?
> >> 
> > Well, I don't know much about the metaprogramming guts of either suite,
> > but I wrote this simple template to create a preallocated vector that I
> > initialize with extract<>(), and exported it using "injected
> > constructors" support:
> >
> > template<typename Container>
> > boost::shared_ptr<Container>
> > create_from_pysequence( object seq)
> > {
> >   boost::shared_ptr<Container> ret( 
> >     new Container(extract<int>(seq.attr("__len__")())));
> >   object seq_i = seq.attr("__iter__")();
> >   for ( typename Container::iterator i = ret->begin(); i != ret->end();
> > ++i) {
> >         *i = extract< typename Container::value_type>( 
> >             seq_i.attr("next")());
> >     }
> >     return ret;
> > }
> 
> What does the constructor injection look like?

See above, and look at boost/libs/python/test/injected.{cpp,py}.

> Ah, OK. Just tried it in 2.3 and got [5,4,3,2,1]. Must be an
> off-by-one error for the None case in my __getitem__.
> 
> >
> > I think that the right way to handle negative step sizes (when you can
> > choose the algorithm) is to use reverse iterators.  The reason is that
> > when provided a negative step, the stop value defaults to "one before
> > the beginning" and the start value defaults to the last element.  So
> > long as you are using the [begin,end) ranges for iterators, the only way
> > to make that work safely is with a reverse iterator and algorithm that
> > carefully accounts for the effects of non-singular step size.

> I don't try to support slices unless the container has random access
> (and then there's no need for a reverse iterator).

Why not?  There is no reason whatsoever not to support every container
with bidirectional iterators.  You can even slice on maps if you want
to, although such a thing has been forbidden for python dict's.

> >
> >> >
> >> > # I think this should raise IndexError; crashes.
> >> > foo[-1:0] = [7, 8, 9]
> >> 
> >> With a real python list, it inserts 7, 8, 9 before the last element:
> >> 
> >> >>> v = [1,2,3,4]
> >> >>> v[-1:0] = [7,8,9]
> >> >>> print v
> >> [1, 2, 3, 7, 8, 9, 4]
> >
> > Yes, that is what happens: performing an insertion before the provided
> > start value.  However, I think that it should be an undefined operation
> > since the expression is utter nonsense.  I've looked at the source code
> > for PyListObject, and I think that this behavior is the result of bounds
> > limiting rather than real error checking.
> 
> I don't understand this. Assigning something into an empty slice in a
> container always performs an insertion. More generally, assigning a
> longer list to a plain slice with fewer elements performs insertion.
> e.g.

No, assigning any sequence to a slice by calling 
object.__setslice__(slice, sequence) replaces the old slice with the
values of the new slice IF the slice uses either -1 or 1 (either
implicitly or explicitly) for its step size.  If the step size is
non-singular, then the sizes of the slice and the sequence must be
identical.

> >>> l = [1,2,3,4]
> >>> l[3:0] = [7,8,9]

Think about what this expression means: Starting at the fourth element
inclusive, and ending at the first element, exclusive with an increment
of forward 1, delete elements and replace them with the elements of the
list [7,8,9].  That's like starting off by calling
std::list::delete(start, stop) with a 'stop' iterator that is not
reachable from 'start'!

The issue isn't that you are performing an insertion after a
well-defined point in the sequence, it is that you are performing a
replacement of an undefined section of the sequence.


> I haven't used Numeric before - is it documented somewhere?

See numpy.sourceforge.net and in Debian the python2.2-numeric,
python2.3-numeric, and python-numeric-tutorial packages.

> >
> > See Python bug# 873305 at http://sourceforge.net/tracker/?group_id=5470
> 
> I don't see any place to enter the bug number - how do I get to see
> bug 873305?

Sorry, that was my fault.  Try this one:
http://sourceforge.net/tracker/index.php?func=detail&aid=873305&group_id=5470&atid=305470

-Jonathan Brandmeyer