What was that web interaction library called again?

[John J. Lee]

Harald Korneliussen <vintermann at gmail.com> writes:

> Hi,
>
> I remember I came across a python library that made it radically
> simple to interact with web sites, connecting to gmail and logging in
> with four or five lines, for example. I thought, "that's interesting,
> I must look into it sometime". Now there's this child I know who asked
> me about programming, especially programs that could do things like
> this, how difficult it was, and so on. I mentioned how I though Python
> was a good intro to programming, and there was a library which was
> perfect for what he wanted.
>
> Only now I've forgotten the name of the library! And try as I might, I
> can't find it with google. I know there are modules for it in the
> standard libraries, but this thing was brilliantly simple in
> comparison. It might have been some sort of research project, I can't
> remember... but perhaps someone here can remind me what it was? If so,
> there may be yet another young python programmer in training :-)

Be warned that all the web scraping tools I know of expose a fairly
leaky abstraction.  One has to know obscure and tiresome details
fairly often.  One common culprit is lack of JavaScript support.
Another is HTML parsing problems.

Still, it's fun when it works.  Test-first development is good thing,
IMHO -- make sure you don't actually have to do network stuff to test
your parsing code, for example, or the delays will get irritating
pretty fast.


John