imaplib: is this really so unwieldy?

[hw]

On 5/25/21 11:38 AM, Cameron Simpson wrote:
> On 25May2021 10:23, hw <hw at adminart.net> wrote:
>> I'm about to do stuff with emails on an IMAP server and wrote a program
>> using imaplib which, so far, gets the UIDs of the messages in the
>> inbox:
>>
>>
>> #!/usr/bin/python
> 
> I'm going to assume you're using Python 3.

Python 3.9.5

>> import imaplib
>> import re
>>
>> imapsession = imaplib.IMAP4_SSL('imap.example.com', port = 993)
>>
>> status, data = imapsession.login('user', 'password')
>> if status != 'OK':
>>     print('Login failed')
>>     exit
> 
> Your "exit" won't do what you want. I expect this code to raise a
> NameError exception here (you've not defined "exit"). That _will_ abort
> the programme, but in a manner indicating that you're used an unknown
> name.  You probably want:
> 
>      sys.exit(1)
> 
> You'll need to import "sys".

Oh ok, it seemed to be fine.  Would it be the right way to do it with 
sys.exit()?  Having to import another library just to end a program 
might not be ideal.

>> messages = imapsession.select(mailbox = 'INBOX', readonly = True)
>> typ, msgnums = imapsession.search(None, 'ALL')
> 
> I've done little with IMAP. What's in msgnums here? Eg:
> 
>      print(type(msgnums), repr(msgnums))
> 
> just so we all know what we're dealing with here.

<class 'list'> [b'']

>> message_uuids = []
>> for number in str(msgnums)[3:-2].split():
> 
> This is very strange. Did you see the example at the end of the module
> docs, it has this example code:
> 
> import getpass, imaplib
> 
>      M = imaplib.IMAP4()
>      M.login(getpass.getuser(), getpass.getpass())
>      M.select()
>      typ, data = M.search(None, 'ALL')
>      for num in data[0].split():
>          typ, data = M.fetch(num, '(RFC822)')
>          print('Message %s\n%s\n' % (num, data[0][1]))
>      M.close()
>      M.logout()

Yes, and I don't understand it.  'print(msgnums)' prints:

[b'']

when there are no messages and

[b'1 2 3 4 5']

So I was guessing that it might be an array containing a single a string 
and that refering to the first element of the array turns into a string 
with which split() can used.  But 'print(msgnums[0].split())' prints

[b'1', b'2', b'3', b'4', b'5']

so I can only guess what that's supposed to mean: maybe an array of many 
bytes?  The documentation[1] clearly says: "The message_set options to 
commands below is a string [...]"

I also need to work with message uids rather than message numbers 
because the numbers can easily change.  There doesn't seem to be a way 
to do that with this library in python.

So it's all guesswork, and I gave up after a while and programmed what I 
wanted in perl.  The documentation of this library sucks, and there are 
worlds between it and the documentation for the libraries I used with perl.

That doesn't mean I don't want to understand why this is so unwieldy. 
It's all nice and smooth in perl.

[1]: https://docs.python.org/3/library/imaplib.html

> It is just breaking apart data[0] into strings which were separated by
> whitespace in the response. And then using those same strings as keys
> for the .fecth() call. That doesn't seem complex, and in fact is blind
> to the format of the "message numbers" returned. It just takes what it
> is handed and uses those to fetch each message.

That's not what the documentation says.

>>     status, data = imapsession.fetch(number, '(UID)')
>>     if status == 'OK':
>>         match = re.match('.*\(UID (\d+)\)', str(data))
> [...]
>> It's working (with Cyrus), but I have the feeling I'm doing it all
>> wrong because it seems so unwieldy.
> 
> IMAP's quite complex. Have you read RFC2060?
> 
>      https://datatracker.ietf.org/doc/html/rfc2060.html

Yes, I referred to it and it didn't become any more clear in combination 
with the documentation of the python library.

> The imaplib library is probably a fairly basic wrapper for the
> underlying protocol which provides methods for the basic client requests
> and conceals the asynchronicity from the user for ease of (basic) use.

Skip Montanaro seems to say that the byte problem comes from the change 
from python 2 to 3 and there is a better library now: 
https://pypi.org/project/IMAPClient/

But the documentation seems even more sparse than the one for imaplib. 
Is it a general thing with python that libraries are not well documented?

>> Apparently the functions of imaplib return some kind of bytes while
>> expecting strings as arguments, like message numbers must be strings.
>> The documentation doesn't seem to say if message UIDs are supposed to
>> be integers or strings.
> 
> You can go a long way by pretending that they are opaque strings. That
> they may be numeric in content can be irrelevant to you. treat them as
> strings.

That's what I ended up doing.

>> So I'm forced to convert stuff from bytes to strings (which is weird
>> because bytes are bytes)
> 
> "bytes are bytes" is tautological.

which is a good thing

> You're getting bytes for a few
> reasons:
> 
> - the imap protocol largely talks about octets (bytes), but says they're
>    text. For this reason a lot of stuff you pass as client parameters are
>    strings, because strings are text.
> 
> - text may be encoded as bytes in many ways, and without knowing the
>    encoding, you can't extract text (strings) from bytes
> 
> - the imaplib library may date from Python 2, where the str type was
>    essentially a byte sequence. In Python 3 a str is a sequence of
>    Unicode code points, and you translate to/from bytes if you need to
>    work with bytes.
> 
> Anyway, the IMAP response are bytes containing text. You get a lot of
> bytes.

Well, ok, but it's not helpful that b is being inserted like everywhere, 
and I have to keep asking myself what I'm looking at because bytes are 
bytes.

Since the documentation is so bad, I had to figure it out by trial and 
error and by printing stuff and making guesses and assumptions.  That's 
just not the way to program something.

> When you go:
> 
>      text = str(data)
> 
> that is _assuming_ a particular text encoding stored in the data. You
> really ought to specify an encoding here. If you've not specified the
> CHARSET for things, 'ascii' would be a conservative choice. The IMAP RFC
> talks about what to expect in section 4 (Data Formats). There's quite a
> lot of possible response formats and I can understand imaplib not
> getting deeply into decoding these.

UTF8 is the default since quite a while now.  Why doesn't it just use that?

>> and to use regular expressions to extract the message-uids from what
>> the functions return (which I shouldn't have to because when I'm asking
>> a function to give me a uid, I expect it to return a uid).
> 
> No, you're asking the IMAP _protocol_ to return you UIDs. The module
> itself doesn't parse what you ask for in the fetch results, and
> therefore it can't decode the response (data bytes) into some higher
> level thing (such as UIDs in your case, but you can ask for all sorts of
> weird stuff with IMAP).

Then its documentation should at least specify what the library does. 
And perhaps it shouldn't specify that some of its functions expect their 
parameters to be strings rather than what other functions return, 
requiring guesswork and conversations of the data because the functions 
kinda aren't compatabile with each other.

> So having passed '(UID)' to the SEARCH request, you now need to parse
> the response.

First I have to guess what the response might be ...  And once I manged 
that, there's still no way to do something with a message by its uid.

>> This so totally awkward and unwieldy and involves so much overhead
>> that I must be doing this wrong.  But am I?  How would I do this right?
> 
> Well, you _could_ get immersed in the nitty gritty of the IMAP protocol
> and the imaplib module, _or_ you could see if someone else has done some
> work to make this easier by writing a higher level library. A search at
> pypi.org for "imap" found a lot of stuff. The package named "imap-tools"
> looks promising. Try that.

Thanks, both the site and the imap-tools package there look interesting.