Word frequencies -- Python or Perl for performance?

[Nick Arnett]

> Well, well, well, long time no see.  Why not just use Verity?  ;-)

I was wondering if that was you -- how many Aahzes could there be in the
world?

> Seriously, for this kind of work, it's quite likely that Perl can be
> coded to be a bit faster than Python, but if you're expecting to need to
> do a lot of iterative work on your algorithms, programmer time will
> count for a lot, and Python will probably win there.

I've actually done a lot of it in Perl already and I'm re-coding it in
Python.  I'm astounded and very happy at how much faster I'm able to code in
Python.  And I can read it a few weeks later without squinting too hard.

Say, how does it look to find a marketing weenie on the Python list?
Actually, I've always coded, I just sometimes manage to supress the urge.
The work I'm doing now is closely related to what I was working on when
Verity lured me, back when the Web was a baby.

> Given that it sounds like you want to create your own inverted word
> index and do some sorting/searching based on word counts, it'll be hard
> to get more bang for the buck than a real database.  Unless you're on a
> shoestring, consider getting a commercial database; you should probably
> also check to see whether MySQL or PostgreSQL will give you better
> performance.

I'm using MySQL; that's what the Perl stuff was talking to.

Say, do you know if anyone has created a Python wrapper for VDK (Verity
Developer Kit, for the rest of you)?  Eventually, when I want to bear the
cost, I'll probably want to use it.  For now, though, I'm trying to stick
with open source tools for anything that gets deployed.

Nick