[Mailman-Users] importing large (1GB) mbox file, hitting a wall here..

[Mark Sapiro]

On 12/28/2017 11:14 AM, Andrew White, PhD wrote:
>    I checked for that- it looks like my problem solving was incomplete. I
>    found an error message when running arch where it was sticking on a bad
>    record - I kept getting "got an unexpected keyword argument 'flags' "
>    (even after using cleanarch on the mbox file), and I think that was the
>    actual problem, not running out of memory.


It looks like we have a bug somewhere. There may be a defective message
in the .mbox, but even so, it should result in a more graceful error report.

Did you get a traceback with the "unexpected keyword argument"
exception. I would like to see a traceback and if possible, the
offending message.

Note that there is a place in the _set_date() function in
Mailman/Archiver/pipermail.py where we are trying to determine the
message's date and if there is no Date: header with a valid date and no
X-List-Received-Date: header with a valid date, we look at a Received:
header and try to extract a date with

    date = floatdate(re.sub(r'^.*;\s*', '',
                            message.get('received', ''), flags=re.S))

but flags=re.S is a valid argument to re.sub. However, you might look in
your mbox for a message without a Date: header.

Also note that cleanarch won't do anything about defective messages. All
it does is look for lines that begin with 'From ' that don't appear to
be mbox message separator 'From ' lines

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan