[Baypiggies] Fwd: Reminder - Tonight: SF Python meet-up featuring Dev Tools for Django and Web2Py

[David Berthelot]

I totally agree with your experience, to squeeze performance you eventually
need to resort to programming which somehow defeats the concept of framework
(which ideally aims at letting you build something without programming). In
my case with Drupal, the code base was so large that I didn't know where to
start (not to mention there is a lot of useless code for my needs but
apparently needed for the sake of generality). However Drupal (and other
frameworks too I suppose) do well for what they were initially intended for
(fancy article/story blogging for Drupal), it's when one tries to customize
to its needs (custom permissions, objects, views) that the limitations
become most noticeable.

In my case, I ended up writing my own performance oriented framework (PyMP)
in Python. I may open source it in the future when I manage to
clean/organize the code a bit more for other people to find reuse value in
it.

This allowed an average request to drop from 10 seconds down to 0.250
seconds (moving from Drupal to PyMP) on a moderately popular website:
http://codercharts.com

On Mon, Jul 18, 2011 at 6:53 PM, Venkatraman S <venkat83 at gmail.com> wrote:

>
>
> On Tue, Jul 19, 2011 at 5:53 AM, David Berthelot <
> david.berthelot at gmail.com> wrote:
>
>> So I generally wouldn't consider the database speed independent of the
>> framework as long as the framework does some automatic database request
>> generation and type handling. This became particularly obvious as the
>> database grew bigger and the framework became slower.
>>
>
> I am a speed maniac, and i like apps to be fast, i am ready to sacrifice
> feature set for good speed; or rather, present the feature set with a few
> more clicks - i.e, if you are showing an item on a page, then i would
> generally like to show everything related to that item in a single page; but
> showing all the related items(which can cause more SQLs and HTTP requests)
> can cause a probable slowdown, i would rather show links which would link to
> the related items (sometimes this approach also leads to a lesser cluttered
> UI - but again, the number of clicks that the user has to do is more [some
> UI experts rate an app based on the number of clicks that a user has to to
> achieve an end result]).
>
> Having said that, its very important to profile an app - in a django app
> that i was developing in my fun time, i happened to see that  there were 40+
> queries on a single page, using DDT i was able to diagnose the SQLs being
> fired, and was able reduce this to just 6 - and out of these 6, 1 sql is for
> hitting the msgs tbl and the other 1 is for checking if the user is logged
> in(unavoidable).
>
> I truly believe in the fact that  "performance is a feature" [ 1<http://www.codinghorror.com/blog/2011/06/performance-is-a-feature.html>]. Also, IMHO, if you can move bulk of the business logic to SQLs (in most
> apps you have to use SQLs), then you can make your code less cluttered and
> also easy to maintain. For eg. in this case [ 2<http://groups.google.com/group/django-users/browse_thread/thread/b2b3aea021c9b7d0/52d1aa80ea7a2dd5>], i moved the filtering to the SQLs; and also even in here, i was firing 3
> queries earlier and was doing some iterations (pain!).
>
> To explain [2], i have 2 types of users: employees and
> general-users(non-emps). I needed to get all items for a given employee or
> customer(non-emp); if the user happened to be employee then get all items
> from his org - so you need to iterate over all emps and get respective items
> for each emp and concatenate the list. If the user was a
> general-user(non-emp), then simply get his items alone.
>
> items = []
> if self.is_superorg_user(): #check user type
>     emps = self.get_all_employees() #get all employees
>     if emps:
>       for e in emps: #iterate over the emps
>         p = Items.objects.filter(created_by=e).filter(deleted=False)  #get
> items by THIS emp
>         if len(p)>0:
>           items.extend(p) #keep concatenating to the orig list
> else: #non-emp user
>   items = Items.objects.filter(created_by=self)
>
> Now, all those SQLs for emp users, can be removed and replaced with:
>
> items = []
> if self.is_superorg_user(): #check user type
>   items =
> Items.objects.filter(created_by__employees__org__in=self.employees_set.all().values_list('org'))
> else:
>   items = Items.objects.filter(created_by=self)
>
> [1] http://www.codinghorror.com/blog/2011/06/performance-is-a-feature.html
> [2]
> http://groups.google.com/group/django-users/browse_thread/thread/b2b3aea021c9b7d0/52d1aa80ea7a2dd5
>
>
> -V
> http://blizzardzblogs.blogspot.com/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/baypiggies/attachments/20110718/bc35d50a/attachment.html>