imaplib: is this really so unwieldy?

[Cameron Simpson]

On 25May2021 10:23, hw <hw at adminart.net> wrote:
>I'm about to do stuff with emails on an IMAP server and wrote a program 
>using imaplib which, so far, gets the UIDs of the messages in the 
>inbox:
>
>
>#!/usr/bin/python

I'm going to assume you're using Python 3.

>import imaplib
>import re
>
>imapsession = imaplib.IMAP4_SSL('imap.example.com', port = 993)
>
>status, data = imapsession.login('user', 'password')
>if status != 'OK':
>    print('Login failed')
>    exit

Your "exit" won't do what you want. I expect this code to raise a 
NameError exception here (you've not defined "exit"). That _will_ abort 
the programme, but in a manner indicating that you're used an unknown 
name.  You probably want:

    sys.exit(1)

You'll need to import "sys".

>messages = imapsession.select(mailbox = 'INBOX', readonly = True)
>typ, msgnums = imapsession.search(None, 'ALL')

I've done little with IMAP. What's in msgnums here? Eg:

    print(type(msgnums), repr(msgnums))

just so we all know what we're dealing with here.

>message_uuids = []
>for number in str(msgnums)[3:-2].split():

This is very strange. Did you see the example at the end of the module 
docs, it has this example code:

import getpass, imaplib

    M = imaplib.IMAP4()
    M.login(getpass.getuser(), getpass.getpass())
    M.select()
    typ, data = M.search(None, 'ALL')
    for num in data[0].split():
        typ, data = M.fetch(num, '(RFC822)')
        print('Message %s\n%s\n' % (num, data[0][1]))
    M.close()
    M.logout()

It is just breaking apart data[0] into strings which were separated by 
whitespace in the response. And then using those same strings as keys 
for the .fecth() call. That doesn't seem complex, and in fact is blind 
to the format of the "message numbers" returned. It just takes what it 
is handed and uses those to fetch each message.

>    status, data = imapsession.fetch(number, '(UID)')
>    if status == 'OK':
>        match = re.match('.*\(UID (\d+)\)', str(data))
[...]
>It's working (with Cyrus), but I have the feeling I'm doing it all 
>wrong because it seems so unwieldy.

IMAP's quite complex. Have you read RFC2060?

    https://datatracker.ietf.org/doc/html/rfc2060.html

The imaplib library is probably a fairly basic wrapper for the 
underlying protocol which provides methods for the basic client requests 
and conceals the asynchronicity from the user for ease of (basic) use.

>Apparently the functions of imaplib return some kind of bytes while 
>expecting strings as arguments, like message numbers must be strings.  
>The documentation doesn't seem to say if message UIDs are supposed to 
>be integers or strings.

You can go a long way by pretending that they are opaque strings. That 
they may be numeric in content can be irrelevant to you. treat them as 
strings.

>So I'm forced to convert stuff from bytes to strings (which is weird 
>because bytes are bytes)

"bytes are bytes" is tautological. You're getting bytes for a few 
reasons:

- the imap protocol largely talks about octets (bytes), but says they're
  text. For this reason a lot of stuff you pass as client parameters are
  strings, because strings are text.

- text may be encoded as bytes in many ways, and without knowing the
  encoding, you can't extract text (strings) from bytes

- the imaplib library may date from Python 2, where the str type was
  essentially a byte sequence. In Python 3 a str is a sequence of
  Unicode code points, and you translate to/from bytes if you need to
  work with bytes.

Anyway, the IMAP response are bytes containing text. You get a lot of 
bytes.

When you go:

    text = str(data)

that is _assuming_ a particular text encoding stored in the data. You 
really ought to specify an encoding here. If you've not specified the 
CHARSET for things, 'ascii' would be a conservative choice. The IMAP RFC 
talks about what to expect in section 4 (Data Formats). There's quite a 
lot of possible response formats and I can understand imaplib not 
getting deeply into decoding these.

>and to use regular expressions to extract the message-uids from what 
>the functions return (which I shouldn't have to because when I'm asking 
>a function to give me a uid, I expect it to return a uid).

No, you're asking the IMAP _protocol_ to return you UIDs. The module 
itself doesn't parse what you ask for in the fetch results, and 
therefore it can't decode the response (data bytes) into some higher 
level thing (such as UIDs in your case, but you can ask for all sorts of 
weird stuff with IMAP).

So having passed '(UID)' to the SEARCH request, you now need to parse 
the response.

>This so totally awkward and unwieldy and involves so much overhead 
>that I must be doing this wrong.  But am I?  How would I do this right?

Well, you _could_ get immersed in the nitty gritty of the IMAP protocol 
and the imaplib module, _or_ you could see if someone else has done some 
work to make this easier by writing a higher level library. A search at 
pypi.org for "imap" found a lot of stuff. The package named "imap-tools" 
looks promising. Try that.

Cheers,
Cameron Simpson <cs at cskk.id.au>