indexing web pages - in python?

[Kevin T. Ryan]

On Apr 18, 8:55 pm, Dan Stromberg <dstromb... at datallegro.com> wrote:
> Are there any open source search engines written in python for indexing a
> given collection of (internal only) html pages?  Right now I'm talking
> about dozens, but hopefully it'll be hundreds or thousands at some point.
>
> I'm thinking some sort of CGI script, with perhaps a cron job that updates
> the indexes.
>
> I'm not particularly looking for something that has a full RDBMS behind
> it - just a file that stores indexes.  I'll go with an RDBMS-based
> solution if I must, but I don't think that's really needed at this point.
>
> TIA

You could try:

http://gnosis.cx/download/indexer.py

There is an extensive write-up by the author at:

http://gnosis.cx/publish/programming/charming_python_15.txt

Might be something you'd be interested in ...