[IronPython] Portable use of pickle.dumps()

[Robert Smallshire]

Hi Michael,

> > I'm trying to get some commercial code for a simple object 
> database we 
> > have written for Python 2.6 to work with IronPython 2.6. In 
> Python 2.6 
> > the return type of pickle.dumps() is str, which is of course a byte 
> > string.  In IronPython 2.6 it is also str, which is of course a 
> > unicode string.  This 'compatibility' is fine until I put those 
> > strings into a database, at which point my interoperability between 
> > CPython and IronPython goes off the rails.
> >
> How is this actually a problem?
> 
> I mean, can you provide a specific example of where a string in 
> IronPython doesn't behave as a byte string in CPython. I'm sure there 
> are such examples, but those may be bugs that the IPy team 
> can fix. In 
> practise I've encountered these problems very rarely.

My opening paragraph may be ambiguously worded - by 'interoperability' I
didn't mean the ability to run the same code unchanged on CPython and
IronPython (I have to change the code anyway to use a different database
adapter) - I meant interoperability between pickles persisted into a
database from both IronPython and CPython.

My basic issue is that the 'str' unavoidably implies certain semantics when
calling .NET APIs from IronPython. These APIs interpret str as text rather
than just bytes, which therefore gets transformed by various text encodings,
such as UTF-8 to UTF-16. Such encodings are undesirable for my pickled data
since the result is no longer necessarily a valid pickle.   I suppose the
intention in Python 3.0 is that 'bytes' doesn't carry any semantics with it,
its just data, which is why pickle.dumps() in Python 3.0 returns bytes
rather than str.

I want to push plain old byte arrays into the database from both CPython and
IronPython, so I can avoid any head-scratching confusion with database
adapters and/or databases inappropriately encoding or decoding my data.

> For example "data = [ord(c) for c in some_string]" has behaved as 
> expected many times for me in IronPython (and could help you turn 
> strings into bytes).

Thanks. I'll try something based on that.

> Is this a theoretical problem at this stage or an actual problem?

Its an actual problem with SQLiteParameter.Value from the SQLite ADO.NET
provider.  I think our original CPython code is a bit sloppy with respect to
the distinction between text strings and byte arrays, so I'll probably need
to tighten things up on both sides.

Would you agree tha using unicode() and bytes() everywhere and avoiding
str() gives code that has the same meaning in Python 2.6, IronPython 2.6 and
Python 3.0?  Do you think this would be a good guideline to follow until we
can leave Python 2.x behind?

Many thanks,

Rob