[Mailman-Users] bin/arch failing

[Mark Sapiro]

Bob wrote:
>
>Everything was going along fine, with daily updates being added to the
>archive. I then wanted to refresh the entire archive and ran bin/arch
>--wipe list1 
>
>it failed. 
>
>Archive articles go from 0000 to 66990
>
>list1.mbox is: 526313754 Aug 21 20:19 list1.mbox
>
>The server NetBSD 3.1, has 2GB memory
>
>bin/arch first went through every article
>
>Week-of-Mon-20070702
>#65136 <20070703134009.74f75a8e at viola.tamara-b.org>
>figuring article archives
>[...]
>
>then after many 
>
>Updating HTML for article 4584
>Updating HTML for article 4585
>Updating HTML for article 4586
>Updating index files for archive [Week-of-Mon-20050321]
>  Date
>  Subject
>  Author
>  Thread
>Computing threaded index
>Updating HTML for article 4587
>Updating HTML for article 4588
>Updating HTML for article 4589
>Updating HTML for article 4590
>Updating HTML for article 4591
>Updating HTML for article 4592
>Updating HTML for article 4593
>
>[...]
>
>Updating HTML for article 9353
>Updating index files for archive [Week-of-Mon-20041122]
>Traceback (most recent call last):
>  File "bin/arch", line 200, in ?
>    main()
>  File "bin/arch", line 190, in main
>    archiver.close()
>  File "/usr/pkg/lib/mailman/Mailman/Archiver/pipermail.py", line 322,
>in close self.update_dirty_archives()
>  File "/usr/pkg/lib/mailman/Mailman/Archiver/pipermail.py", line 538,
>in update_dirty_archives self.update_archive(i)
>  File "/usr/pkg/lib/mailman/Mailman/Archiver/HyperArch.py", line 1121,
>in update_archive self.__super_update_archive(archive)
>  File "/usr/pkg/lib/mailman/Mailman/Archiver/pipermail.py", line 439,
>in update_archive self.__set_parameters(archive)
>  File "/usr/pkg/lib/mailman/Mailman/Archiver/pipermail.py", line 354,
>in __set_parameters firstdate = self.database.firstdate(archive)
>  File "/usr/pkg/lib/mailman/Mailman/Archiver/HyperDatabase.py", line
>209, in firstdate self.__openIndices(archive)
>  File "/usr/pkg/lib/mailman/Mailman/Archiver/HyperDatabase.py", line
>240, in __openIndices self.__closeIndices()
>  File "/usr/pkg/lib/mailman/Mailman/Archiver/HyperDatabase.py", line
>265, in __closeIndices index.close()
>  File "/usr/pkg/lib/mailman/Mailman/Archiver/HyperDatabase.py", line
>187, in close fp.write(marshal.dumps(self.dict))
>MemoryError
>


This problem is mentioned in the description of the -e option of
bin/arch --help.


>I can successfully run about the last 10,000 
>
>bin/arch -s 50000 -e 66990 list1
>
>But that is about 1/7th of the archive
>Can I produce a complete archive in chunks of 10,000? How?


bin/arch --wipe -e 99999 list1
bin/arch -s 10000 -e 19999 list1
bin/arch -s 20000 -e 29999 list1
...
bin/arch -s 60000 list1

Note that --wipe is specified only on the first command.

Also, it is a good idea to stop Mailman during this process. The reason
is that if a new message arrives and is archived say between the
second and third commands, it will be assigned message number 20000 in
the new archive, but it will be message number 66991 in the .mbox.
This will guarantee that if you ever rebuild the archive again, all
message numbers from 20000 on will change, thus invalidating any saved
archive URLs.

Finally, it is a good idea to first check the .mbox file with
bin/cleanarch to make sure there are no unescaped From_ lines in
message bodies.

-- 
Mark Sapiro <msapiro at value.net>       The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan