BeautifulSoup doesn't work with a threaded input queue?

MRAB python at mrabarnett.plus.com
Sun Aug 27 16:50:20 EDT 2017


On 2017-08-27 21:35, Christopher Reimer via Python-list wrote:
> On 8/27/2017 1:12 PM, MRAB wrote:
> 
>> What do you mean by "queue (random order)"? A queue is sequential 
>> order, first-in-first-out. 
> 
> With 20 threads requesting 20 different pages, they're not going into
> the queue in sequential order (i.e., 0, 1, 2, ..., 17, 18, 19) and
> coming in at different times for the parser worker threads to get for
> processing.
> 
> Similar situation with a list but I sort the list before giving it to
> the parser, so all the items are in sequential order and fed to the
> parser one at time.
> 
What if you don't sort the list? I ask because it sounds like you're 
changing 2 variables (i.e. list->queue, sorted->unsorted) at the same 
time, so you can't be sure that it's the queue that's the problem.



More information about the Python-list mailing list