web crawler in python or C?

[dfj225 at gmail.com]

I think something that may be even more important to consider than just
the pure speed of your program, would be ease of design as well as the
overall stability of your code.

My opinion would be that writing in Python would have many benefits
over the speed gains of using C. For instance, you crawler will have to
handle all types of input from all over the web. Who can say what types
of malformed or poorly writen data it will come across. I think it
would be easier to create a system to handle this type of data in
Python than in C.

I don't want to pigeon-hole your project, but if it is for any use
other than a commercial product, I would say speed would be a concern
lower on the list than accurracy or time to develop. As others have
pointed out, if you hit many performance barriers chances are the
problem  is the algorithm and not Python itself.

I wish you luck and hope you will experiment in Python first. If your
crawler is still not up to par, at the very least you might come up
with some ideas for how Python could be improved.