Asyncio Queue implementation suggestion

[Léo El Amri]

Hello Alberto,

I scrambled your original message a bit here.

> Apparently asyncio Queues use a Linux pipe and each queue require 2 file
> descriptors. Am I correct?

As far as I know (And I know a bit about asyncio in CPython 3.5+)
asyncio.queues.Queue doesn't use any file descriptor. It is implemented
using futures which are themselves tightly implemented with the inner
workings of the event loop in mind.

> I wrote a daemon in Python 3 (running in Linux) which test many devices
> at the same time, to be used in a factory environment. This daemon
> include multiple communication events to a back-end running in another
> country. I am using a class for each device I test, and embedded into
> the class I use asyncio. Due to the application itself and the number of
> devices tested simultaneously, I soon run out of file descriptor. Well,
> I increased the number of file descriptor in the application and then I
> started running into problems like “ValueError: filedescriptor out of
> range in select()”. I guess this problem is related to a package called
> serial_asyncio, and of course, that could be corrected. However I became
> curious about the number of open file descriptors opened: why so many?

I don't know about a CPython internal called serial_asyncio.

I wonder whether the issue is coming from too much file descriptors...
1. If so, then I assume the issue you encounter with the selector is
coming from the number of sockets you open (Too many sockets).
2. Otherwise it could come from the fact that some sockets gets
destroyed before the selector gets a chance to await on the file
descriptors.

Without a complete traceback I can't be 100% certain, but the error
message you get looks like a EBADF error. So the second hypothesis is my
preferred.

Have you tried enabling asyncio's debug mode ? I hope it can give you
more information on why this is occurring. I believe it's a bug, not a
limitation of CPython.

- Léo