[omaha] GAE support for web2py - Some corrections

[Matthew Nuzum]

Sounds interesting. I deployed a few apps to GAE, though not with web2py,
and I found one challenging issue: Bigger apps are unloaded from memory
rather quickly and can take 6 - 8 seconds to be loaded up. Therefore if
your app doesn't get frequent traffic (one page view per min or so) then
the first load will have a white-screen pause while it is loaded back into
memory.

It doesn't sound like much but it can be quite annoying. And because it
affects the first page view it happens before any images or css are loaded,
so you get nothing while you wait.

Out of curiosity, is this still an issue or is it resolved?

Also, how does someone get access to the googleSQL storage? Is it the same
thing as the Cloud SQL at https://developers.google.com/cloud-sql/ ? My
googling takes me to app engine for business which is no longer offered.

On Thu, Feb 23, 2012 at 2:26 PM, Tim Alexander <dragonfyre13 at gmail.com>wrote:

> Wanted to post to the list, since I was inaccurate on a few things in
> talking about GAE support in web2py. Some of the stuff I was thinking was
> surrounding the googleSQL engine, and some of it was workarounds that I
> thought were transparent (below). Here's a synopsis of
> restrictions/workarounds/alternatives on GAE. Note, everything else in
> web2py works as expected, other than the restrictions, most have relatively
> straightforward workarounds (posted to the web2py mailing list for most of
> them, some in the book itself, but most DB related in the
> web2pyslices.comlink directly below)
>
> First, the good stuff: In 1.69.1 (in 2009) "workarounds" were added to
> web2py so that what can't be done in BigTable itself is done at the web2py
> layer level. For example, joins, OR, orderby (including complex orderby),
> LIKE, datetime manipulations, etc. Here's a great
> page<http://www.web2pyslices.com/slices/take_slice/16>on those, and
> they work quite well in my limited experience and from what I
> hear. I believe it's not in the book because you have to be careful of
> using these methods since they can potentially be quite inefficient
> (performing evaluations in python of large data sets, rather than a
> database).
>
> Straight from the book, however:
>
> *GAE also supports a Google SQL database (compatible with MySQL) and a
> Google NoSQL (referred to as "Datastore").**
> **web2py supports both. If you wish to use Google SQL database follow the
> instructions on Chapter 6. This section assues you will be using the Google
> Datastore.**
> **The GAE platform provides several advantages over normal hosting
> solutions:*
>
>
>   - *Ease of deployment. Google completely abstracts the underlying
>      architecture.*
>      - *Scalability. Google will replicate your app as many times as it
>      takes to serve all concurrent requests.*
>      - *One can choose between a SQL and a NoSQL database (or both
>      together).*
>
> *But also some disadvantages:*
>
>
>   - *No read or write access to the file system.*
>      - *No HTTPS unless you use the appspot.com domain with a Google
>      certificate.*
>
> *and some Datastore specific disadvantages:*
>
>
>   - *No typical transactions.*
>      - *No complex datastore queries. In particular there are no JOIN,
>      LIKE, and DATE/DATETIME operators.*
>      - *No multiple OR sub-queries unless they involve one and the same
>      field.*
>      - *Because of the readonly filesystem, web2py cannot store sessions,
>      error tickets, cache files and uploaded files in the filesystem;
> they must
>      be stored in the datastore and not in the filesystem.*
>
> Also from another section of the book:
>
> *Google NoSQL (Datastore) does not allow joins, left joins, aggregates,
> expression, OR involving more than one table, the like operator and search
> in "text"" fields. Transactions are limited and not provided automatically
> by web2py (you need to use the Google API run_in_transaction which you can
> look up in the Google App Engine documentation online). Google also limits
> the number of records you can retrieve in each one query (1000 at the time
> of writing). On the Google datastore record IDs are integer but they are
> not sequential. While on SQL the "list:string" type is mapped into a "text"
> type, on the Google Datastore it is mapped into aListStringProperty.
> Similarly "list:integer" and "list:reference" are mapped into
> "ListProperty". This makes that searches for content inside these fields
> types are more efficient on Google NoSQL than on SQL databases.*
>
>
> Note, joins and many to many relationships be worked around using
> tagging<http://web2py.com/books/default/chapter/29/6#Many-to-many>,
> and the "OR" restriction is worked around in the current DAL code by
> essentially doing it programmatically. Text field search IS supported in
> two ways, *kinda*. See this
> post<http://www.billkatz.com/2009/6/Simple-Full-Text-Search-for-App-Engine
> >for
> the idea behind how it's done. Essentially, it boils down to storage
> of
> ngrams into a stringlistproperty<
> http://code.google.com/appengine/docs/python/datastore/typesandpropertyclasses.html#StringListProperty
> >object,
> which actually works quite well on GAE (been pleased with this
> method for other stuff). If you just want simple, and not neccesarily
> performant, this works fine too, though since it's run directly in web2py
> it is more expensive (this is the semi-official way of doing in web2py):
> db(...).select().find(lambda row: row.find('substring')>=0)
>
> Eli, you were interested specifically in OR queries, here's the method as
> on
> the slices page <http://www.web2pyslices.com/slices/take_slice/16> (with a
> complex orderby thrown in). You can see the computation is done in web2py,
> rather than the DB:
> on RDBS
>
> rows = db((purchase.buyer==kenny)|(purchase.buyer==cartman))\
>         (purchase.buyer==buyer.id).select(orderby=buyer.name)
>
> on RDBS and GAE
>
> rows = (db(purchase.buyer==kenny).select()|db(purchase.buyer==cartman)\
>       .select()).sort(lambda row:row.buyer.name)
> print rows
> _______________________________________________
> Omaha Python Users Group mailing list
> Omaha at python.org
> http://mail.python.org/mailman/listinfo/omaha
> http://www.OmahaPython.org
>



-- 
Matthew Nuzum
newz2000 on freenode, skype, linkedin and twitter

♫ You're never fully dressed without a smile! ♫