Python example: possible speedup?

[Cedric Adjih]

Hrvoje Niksic <hniksic at srce.hr> wrote:
> Doug Hellmann <doughellmann at home.com> writes:

>> > Thanks for the suggestion.  However, I do install new packages
>> > often, so that's not really an option.
>> 
>> Probably not every time you run your app, though.

> I use the application to see which packages waste too much space, so
> that I know which ones to remove.  Not to mention that I run nightly
> updates of my Debian system.  It is practically *guaranteed* that the
> files will have changed between two runs of the program.
>
> Sorry.  I like the idea of pickling/marshalling, and I'd love to be
> able to use it here.  But it can't work that way.

  Maybe you can update the python structures at night too 
(the same way the 'locate' database is built at night,
because a 'find' on the whole system is too slow).

>> You could parse the data, pickle the results, and keep up with the
>> mod times on the two files.  If the source data is newer than the
>> pickle database you have to update the pickle data.
>
> The problem is that to "update" the pickle data (which I'd have to do
> every time, as explained above), I have to parse the text files.  And
> it's the parsing that's slow, not the post-processing.  If I need to
> parse them every time, then I don't need pickling.

  Yes, the idea was to keep a cache of the pythonic database, so
whenever it has changed, you rebuild it, and save it, for reusing
it later.
  Doing something clever would involve using "diff" for instance, but
is probably more complex than making a C module for Python :-).

-- Cedric Adjih