[Chicago] AppEngine Bulk Loading

[Feihong Hsu]

Argh. I can't use "line.encode('ascii', 'replace')" because many of
the strings in my app are ALL Unicode, and there would be no
distinction between them anymore.

Well, the BulkLoad class is just another WSGI app, so I can write my
own custom bulk loading app and it shouldn't perform any worse,
right?

Anyway, I have about 45k entities to load. Is there anyone with
experience bulk loading a large number of items to AppEngine? Do I
even want to know how long that's gonna take?


--- Kumar McMillan <kumar.mcmillan at gmail.com> wrote:

> On Mon, May 12, 2008 at 11:20 AM, Feihong Hsu
> <hsu.feihong at yahoo.com> wrote:
> > My cookie value doesn't look like that on the dev server, but I
> >  assume that's what it is when you try it on production.
> >
> >  Thanks for anticipating my Unicode problem there, it just so
> happens
> >  that every single one of my entities has Unicode strings. I just
> >  upgraded to version 1.0.2 of the SDK and they still haven't
> fixed
> >  this annoying problem.
> 
>
http://code.google.com/p/googleappengine/issues/list?q=unicode&can=2
> 
> everyone should star these issues ;)
> 
> 
> in the meantime, you'd have to patch bulkload_client to use the
> unicode_csv_reader hack:
> http://docs.python.org/lib/csv-examples.html
> 
> and then see if it works.  if you run into the same problem I was
> having with  datastore_types.ToPropertyPb() then I suppose you'll
> have
> to give up and create a wrapper that does line.encode('ascii',
> 'replace').  That would at least give you some data to work with in
> your app.  Then you can reload it when they fix Unicode.
> 
> 
> K
> _______________________________________________
> Chicago mailing list
> Chicago at python.org
> http://mail.python.org/mailman/listinfo/chicago
> 



      ____________________________________________________________________________________
Be a better friend, newshound, and 
know-it-all with Yahoo! Mobile.  Try it now.  http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ