web crawler in python or C?

[Ravi Teja]

This is following the pattern of your previous post on language choice
wrt. writing a mail server. It is very common for beginers to over
emphasize performance requirements, size of the executable etc. More is
always good. Right? Yes! But at what cost?

The rule of thumb for all your Python Vs C questions is ...
1.) Choose Python by default.
2.) If your program is slow, it's your algorithm that you need to check
first. Python strictly speaking will be slow because of its dynamism.
However, most of whatever is performance critical in Python is already
implemented in C. And the speed difference of well written Python
programs with properly chosen extensions and algorithms is not far off.
3.) Remember that you can always drop back to C where ever you need to
without throwing all of your code. And even if you had to, Python is
very valuable as a prototyping tool since it is very agile. You would
have figured out what you needed to do by then, that rewriting it in C
will only take a fraction of the time compared to if it was written in
C directly.

Don't even start with asking the question, "is it fast enough?" till
you have already written it in Python and it turns out that it is not
running fast enough despite correctness of your code. If it does, you
can fix it relatively easily. It is easy to write bad code in C and
poorly written C code performance is lower than well written Python
code performance.

Remember Donald Knuth's quote.
"Premature optimization is the root of all evil in programming".

C is a language intended to be used when you NEED tight control over
memory allocation. It has few advantages in other scenarios. Don't
abuse it by choosing it by default.