Question concerning this list

[Thomas Ploch]

Steven D'Aprano wrote:
> On Sun, 31 Dec 2006 02:03:34 +0100, Thomas Ploch wrote:
> 
>> Hello fellow pythonists,
>>
>> I have a question concerning posting code on this list.
>>
>> I want to post source code of a module, which is a homework for
>> university (yes yes, I know, please read on...).
> 
> So long as you understand your university's policy on collaborations.

Well, collaborations are wanted by my prof, but I think he actually
meant it in a way of getting students bonding with each other and
establishing social contacts. He just said that he will reject copy &
paste stuff and works that actually have nothing to do with the topic
(when we were laughing, he said we couldn't imagine what sometimes is
handed in).

>> It is a web crawler (which I will *never* let out into the wide world)
> 
> If you post it on Usenet, you will have let it out into the wide world.
> People will see it. Some of those people will download it. Some of them
> will run it. And some of them will run it, uncontrolled, on the WWW.
> 
> Out of curiosity, if your web crawler isn't going to be used on the web,
> what were you intending to use it on?

It's a final homework, as I mentioned above, and it shouldn't be used
anywhere but our university server to test it (unless timing of requests
(i.e. only two fetches per second), handling of 'robots.txt' is
implemented). But you are right with the Usenet thing, havn't thought
about this actually, so I won't post the whole portion of the code.

>> which uses regular expressions (and yes, I know, thats not good, too).
> 
> Regexes are just a tool. Sometimes they are the right tool for the job.
> Sometimes they aren't.

Alright, my prof said '... to process documents written in structural
markup languages using regular expressions is a no-no.' (Because of
nested Elements? Can't remember) So I think he wants us to use regexes
to learn them. He is pointing to HTMLParser though.

>> I have finished it (as far as I can), but since I need a good mark to
>> actually finish the course, I am wondering if I can post the code, and I
>> am wondering if anyone of you can review it and give me possible hints
>> on how to improve things.


> It probably isn't a good idea to post a great big chunk of code and expect
> people to read it all. If you have more specific questions than "how can
> I make this better?", that would be good. Unless the code is fairly
> short, it might be better to just post a few extracted functions and see
> what people say about them, and then you can extend that to the rest of
> your code.

You are probably right. For me it boils down to these problems:
- Implementing a stack for large queues of documents which is faster
than list.pop(index) (Is there a lib for this?)
- Getting Handlers for different MIME/ContentTypes and specify callbacks
only for specific Content-Types / MIME-Types (a lot of work and complex
checks)
- Handle different encodings right.

I will follow your suggestions and post my code concerning specifically
these problems, and not the whole chunk.

Thanks,
Thomas