[Python-ideas] New imaplib implementation for Python 3.2+ standard library

[Menno Smits]

(I'm replying via Google Groups because I just joined and don't have
this thread in my email inbox. It's being a bit flaky so apologies if
this comes though twice).

On Jul 26, 5:54 pm, Maxim Khitrov <m... at mxcrypt.com> wrote:
>
> One of the things I'm trying to address with my library is strict
> adherence to the current version of the IMAP4 protocol. The other is
> performance; hence the implementation of extensions such as SASL-IR,
> IDLE, non-synchronizing literals, multiappend, and compression.
>
> On the performance side, if you have an application that's trying do
> some sort of processing of a 6 GB mailbox with 700,000 messages in it,
> executing a separate FETCH command for each message will take you a
> week to finish. If you try to be clever and FETCH 1000 messages at a
> time, for example, you'll quickly run into a few problems:
>
> ...
> All are interface design problems, which are inherited by IMAPClient.
>
> - Max
>
> P.S. It is not my intention to discourage the use of IMAPClient in any
> way. Its existence is a good thing for 99% of the users, because it
> does address a number of key imaplib issues with just the response
> parser and a UTF-7 codec. My point is that there are real-world use
> cases out there that cannot be handled by imaplib or IMAPClient, and
> for those, I'm offering my library as a more general solution that
> should satisfy the remaining 1% :)

As the maintainer of IMAPClient, I thought I'd weigh in.

I've had a quick look at imaplib2[1] and I must say it's a solid piece
of work. The number of IMAP extensions that it covers is impressive,
the
iterative streaming of fetch responses is great and the way that
concurrent and async commands are handled is quite elegant.

Max is correct about the limitations of imaplib (and therefore
IMAPClient). As with many Python libraries (eg. poplib, email,
smtplib),
data is loaded into memory without the option of streaming (say via
generators) and imaplib is not designed with asynchronous handling in
mind. This simplicity has benefits in terms of the implementation and
usage of the API but it can also lead to problems. I have considered
restructuring IMAPClient so that it no longer depends on imaplib and
so
free it from those inherent limitations. This may still happen.

imaplib2 makes good use of the features available in modern versions
of
Python. On the other hand, one of my goals with IMAPClient is to
support
a wide range of Python versions - many newer Python niceties are not
used. Python 3 support for IMAPClient is definitely coming but a new
addition to our family at home as slowed that effort down somewhat.

I have never considered pushing for IMAPClient to be included as part
of
the stdlib given that it is fairly easy to install 3rd party packages
these days, but I wouldn't be against it. Obviously Python 3 support
would have to come first.

I think imaplib2 is a very capable IMAP client library and the Python
community could only benefit from having something like it in the
standard library (on the proviso, as Brett mentions, that the Python
community supports the library by using it widely).

Here's a few comments about imaplib2 from my own biased perspective:

It requires too much effort on behalf of the caller. Your example.py
highlights how datetimes are returned as strings that need to be
converted to real datetimes and FETCH response keys need to be
uppercased to ensure consistency. The need to jump through the same
non-trivial hoops each time I used imaplib was one of the frustrations
that led to the creation of IMAPClient. Please consider having
imaplib2
do a little more work so that every user doesn't have to.

Similarly, UID support could be better. IMAPClient has a boolean
attribute which lets you select whether you want UIDs to be
transparently used for future commands. Having to specify whether you
want UID support enabled on each call is a little clumsy. It's
unlikely
that a user of imaplib2 would want to toggle between using UIDs and
not
on every call.

This has already been mentioned but imaplib2 won't get accepted into
the
stdlib if you don't conform to PEP 8. Those tabs have to go.

How much testing has imaplib2 seen against real IMAP implementations?
Throughout IMAPClient's history its users have found many unexpected
behaviours in various popular IMAP implementations. Those discoveries
have lead to updates to IMAPClient's code and tests (this is the
"battle-tested" aspect that Michael refers too). On top of its unit
tests, IMAPClient has a fairly extensive live test script that can be
run (destructively) against a real IMAP account. I have test accounts
with a number of different IMAP implementations which I regularly test
IMAPClient against. A set of "live" tests is invaluable for testing
new
features and avoiding regressions between versions. It would be
interesting to see what problems you find if you set up something
similar for imaplib2.

Anyway, I wish you all the best with your project.

Regards,
Menno

[1] - Are you aware there's already another project with the same
name?
http://www.janeelix.com/piers/python/imaplib2.html