Right tools for python info space mapping project?

[oivvio at polite.se]

Hi all.

I've been thinking of writing an info space mapping program for some
time. (Check out http://www.geog.ucl.ac.uk/casa/martin/atlas/atlas.html
for some
examples of info space mapping.)

I want to retrieve material from the net (my first priority is newspaper
sites), store it in a database, calculate relationships between texts
and make  graphical representations of theese relationships.

I hope that this will mostly consist of gluing together existing
open sourced software. I'll be using python for this gluing work.

This is what I need:

1. RDBMS
MySQL, or PostgreSQL. Which one has better python bindings?

2. Full text indexing.
As neither of the free RDBMS have full text indexing this will have to
been done separately. Sourceforge hosts Wordindexer (a perl thingy that
works with MySQL), mifluz (a c++ word indexer), and pyMiflus (python
bindings for mifluz). What are you're experiences with these and
similar tools.

3. Parsing websites.
I need a way to describe the structure of a website (what's to be
downloaded and what's not.) I haven't found anything in this
category. Have you?

4. Calculating relationships between texts based and word frequencies.
I guess this will be a work for NumPy

5. Making the pictures.
There seems to be about a million graphics/plotting packages for
python. I'm looking for something that has a lot of capabilities for
making the pictures very pretty. ImageMagick, vtk, PIL or what?

All ideas and  pointers are welcome.

regards oivvio


Sent via Deja.com
http://www.deja.com/