[IronPython] LINQ from IronPython

[Dino Viehland]

Jeff wrote:
> This is one of those things that I've been pondering for a while, so I
> thougt I'd throw it out there and see what everyone else thinks (I
> fully expect to hear that I'm crazy...). I'd like to be able to query
> an existing data model (Entity Framework in this case, but that's not
> important) from IronPython without pulling all of the data to the
> client. By my read there are two necessary things and one nice-to-have
> to enable this.

I'd say this is a pretty good write up of the issues and potential 
solutions.  Not really crazy at all.

> 
> This is just a bunch of random thoughts about how to get LINQ support
> into IronPython; I know nothing about the details of the IronPython
> compiler, so I'm making no assumptions about the difficulty of any of
> this.
> 
> Extension Methods
> --
> The biggest hurdle (but probably the most fruitful) is enabling
> extension methods. Unfortunately, I don't know enough about how
> IronPython looks up methods on .NET classes to know how easy this is,
> so I'll just ramble about possible syntaxes and assume the lookups
> will work.
> 
> The first option is identical to C#:
>    from System import Linq    # or...
>    import System.Linq
> 
> Either form pulls in all extension methods in all public, static types
> in that namespace.
> 
> The second option is similar to C#, but requires knowing the extension
> class as well:
>    from System.Linq import Queryable
> 
> This form would only pull in extension methods defined in the Queryable
> class.
> 
> I prefer the second option over the first ('explicit is better than
> implicit'), but both have problems in not being completely clear what
> is happening.
> 
> Another option is to be completely explicit:
>    clr.Extend(myObj, System.Linq.Queryable)

Personally I like the 2nd option.  I think only the 2nd and 3rd are
really viable.  The one important thing to keep in mind here is that
importlib may become the default importer in the future.  So any hooks
here need to be based off of the processing of what the import machinery
brings back - not something buried deep in the import machinery.  The 2nd
form means we only need to check the type going back vs. it's entire 
namespace which will be more expensive.

The 3rd option could also be done w/o myObj, e.g. you just say bring
in this type or namespace.  It could also be global (not very nice) or
it could be local to the module like the other forms presumably are.

>From an implementation perspective the real difficulty here is making
our get members be isolated across modules while still enabling sharing
of our cached rules. 

> 
> It's nice and explicit about what is happening, but could get tedious
> quickly.
> 
> Expression Trees (C# lambdas)
> --
> Trying to do
>    Queryable.Where(entities.Customers, lambda c: c.Id % 2 == 0))
> gives (not surprisingly)
>    The type arguments for method 'Where' cannot be inferred from the
> usage.
> 
> Putting in the generic argument manually
>    Queryable.Where[Customer](entities.Customers, lambda c: c.Id % 2 ==
> 0))
> I get (again, not surprisingly)
>    `expected Expression[Func[Course, int, bool]], got function`
> 
> So, at the very least lambdas would have to be convertible to
> Expression<> objects. I know the DLR uses expression trees internally,
> but I don't know if they're available at the proper time for that
> conversion to be possible. The type inference doesn't seem anymore
> difficult than what's already done, but who knows...

We certainly have the raw AST available to us at all times except for
when we've pre-compiled code.  We don't expose this directly today just
because it's a big implementation detail to bleed out.  Exposing a 
conversion of functions to Experssion<T> though might be a really good
way to expose it w/o exposing all of the implementation details.

The biggest problem w/ exposing our AST though is can anyone else do
something useful with it?  If they're just going to compile the AST
and run it then that's fine.  But if they want to take the AST and
send it off to the database as a query things aren't going to work
so well at this point simply because they won't recognize the dynamic
nodes.  There's also some situations where we open code to the dynamic
nodes as call sites then things are really confusing.  This may
be fine for this case where the conversion is explicit and the user
can make sure it works.

> 
> Generator Expressions as LINQ exprssions
> --
> With extension methods and expression trees, the Q in LINQ is handled;
> now the LIN ('Language INtegrated') needs to be filled in.
> 
> Most of the LINQ queries I do are of the form:
>    from customer in entities.Customers
>    where customer.Name == name
>    select customer
> 
> This just so happens to look a lot like
>    customer for customer in entities.Customers if customer.Name == name
> 
> i.e. a generator expression.
> 
> Generator expressions and LINQ queries have a lot in common - chief
> amongst them is that they're both lazy. Would it be possible, if
> `entities.Customers ` implements IQueryable, for a generator
> expression to be converted into a LINQ query? Would it even be
> desirable?
> 
> The alternative is
>    entities.Customers.Where(lambda c: c.Name == name)
> 
> which is perfectly fine, but something more would be nice.

This is actually a really cool idea.  But because it'd occur implicitly
it has the same problems as the function ASTs do but they might otherwise
break working code.  

Maybe we could try and create another AST with faked out expressions 
which indicate our intent without specifying exactly how to do it.  But
the expression trees are really particular about being valid so we'd
probably need to know the type of entities first and then infer the 
types from there.  This would probably be the trickiest part.

> --
> 
> One thing all of this is that it doesn't change the syntax of Python
> at all, just the semantics. I'm not sure if that's good or bad,
> though.
> 
> Thoughts?
> 
> - Jeff
> _______________________________________________
> Users mailing list
> Users at lists.ironpython.com
> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com