Queue module and Python Documentation Rant

[Roger Binns]

Bart Nessux wrote:
> How can one view the contents of a queue?

You are over thinking the whole thing :-)

Especially note that the Queue is intended to be used in threaded
programming (multi-producer, multi-consumer) which the doc clearly
states.

> I'd like to verify that the queue has the same number of objects
> as the list that has been put into it.

There is a qsize() method clearly documented.  What is wrong with
that?

> Also, should one think of a queue as static or dynamic (on the put
> side)?

The constructor documentation clearly describes how it behaves.
You can limit the number of items in the Queue at which point
new puts will block, or you can have the default behaviour which
allows unlimited items in the queue.

> Do you put stuff in it once and then get from it until it's empty
> or do you put stuff in it a bit at a time?

You put items into the Queue and get items out.  Each item is an
object of any type.  The Queue does not split up items.  For example
if you place a string into the queue, you cannot retrieve just part
of the string first and another part later.  You only get the whole
string out.

> Also,
> does the queue actually get smaller as the threads feed from it or does
> it stay the same size and remember what stuff has been gotten from it
> already?

It is a classic queue in the computer science sense.  Objects can be added
(via the put method) and taken off (by the get method).  The size is the
number of objects in the queue.  The order is FIFO (first in first out).
Contrast with a stack which is LIFO (last in is the first out).

The "staying the same size" part of your comment usually refers to
another data type known as a circular buffer (or circular queue).

I would strongly recommend getting a copy of the Python Cookbook
and reading it.  The contents are also available online but the
dead tree version is a better read.

> Is this an appropriate way to create a queue of 100,000 items?
>
> url_queue = Queue.Queue(0)
> for url in urls:
>      url_queue.put(url)

Yes.  However in most code you are unlikely to do that.  In general
you will have some code producing items, and other code consuming
them.  Another thread in this group had an example.  One piece of
code was producing IP addresses to scan, and a seperate piece of
code (for example a worker thread) would take one work item and
do the scanning.

Consequently the number of items would be small since you would expect
the rate at which items are produced to be a similar order of magnitude
to the rate at which they are consumed.

> I ask all of these questions because I find the Python documentation
> lacking, to say the least.

The documentation for this class is actually short and sweet and I
don't think it could be improved.  How would you improve it?

It is however missing an example which would make things a lot clearer
especially for people who aren't used to standard programming idioms.
The tutorial is a little brief but does cover data structures:
http://docs.python.org/tut/tut.html

> PHP especially.

Pretty much anyone who has done PHP in anger raves about their docs (me
included).  Not only is the meta-data really good (eg which version the
item was introduced in etc), but the user contributions are what makes
the big difference.  Also as far as I can tell, every single page includes
an example.

There was a discussion here about doing the Python docs in a similar fashion
(especially the user annotations) a few weeks ago.  As far as I could tell,
the results were that anyone who wanted to submit changes had to do it via
SourceForge bugs and wait a fair while (months) for the changes to appear,
or construct a wiki that is seperate from the docs.

> Python should be ashamed of its module documentation

The documentation is fine.  It is pretty standard for docs constructed
"cathedral" style.  I do wish it was more bazaar style like the PHP
ones, but I don't have the time or inclination to help put the infrastructure
in place.  Do you?

Roger