Program works great, except under less, cron or execl (Unicode?)

[Diez B. Roggisch]

Sam wrote:

> I have a program which works great when run from the command line.
> 
> But when I run it combined with something else such as:
> - piping it through less
> - cron
> - execl (i.e. calling it from another python program)
> 
> it gives me a unicode error
> 
>  File "../myparser.py", line 261, in set_attributes
>     print "self.atd['Name'] is: ", self.atd['Name']
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xeb' in
> position 7: ordinal not in range(128)
> 
> I'd post the whole program here, except it involves weird Unicode
> strings.
> 
> I could probably change the program to get it working under less/cron/
> etc.
> 
> But I'd rather understand exactly what the issue is.  Why does it work
> fine when run directly from the command line, but not otherwise?

Most probably because when to running directly inside a terminal, it gets
it's stdin/stdout as pipes - and python can't attempt to guess the proper
encoding on that, as it does on a terminal.

And thus, when you print unicode to the pipe, it can't decide which encoding
to use.

To circumvene this, try & wrap stdout into a codecs-module wrapper with a
proper encoding applied (e.g. utf-8).

You might make that conditionally based on the sys.stdout.encoding-variable
being set or not, albeit I'm not 100% sure to what it actually gets set
when used in a subprocess. But this should give you the idea where to look.



Diez