[Mailman-Users] importing large (1GB) mbox file, hitting a wall here..

[Andrew White, PhD]

   I checked for that- it looks like my problem solving was incomplete. I
   found an error message when running arch where it was sticking on a bad
   record - I kept getting "got an unexpected keyword argument 'flags' "
   (even after using cleanarch on the mbox file), and I think that was the
   actual problem, not running out of memory. I ran it last night removing
   that record, and it worked without batching as long as I didn't include
   that batch of records only about .3% of the file..

   At 09:33 AM 12/28/2017, you wrote:

     On 12/27/2017 08:08 PM, Andrew White, PhD wrote:

     >    I then ran bin/arch --wipe dbt-l_pdbti.org. When I
     >    checked the archives, only about 11,000 messages were imported. I
     saw in
     >    the arch help file there can be memory issues, and so to run things
     in
     >    chunks. So, I did this:
     >
     >    bin/arch ---wipe q -s 0 e 10000 dbt-l_pdbti.org
     >    bin/arch -q -s 10001 e 20000 dbt-l_pdbti.org
     >    bin/arch  q -s 20001 e 30000 dbt-l_pdbti.org
     >    bin/arch  q -s 30001 e 40000 dbt-l_pdbti.org
     >
     >    So when I do this, each piece works, but each piece overwrites the
     >    previous- in other words, rather than each chunk adding into the
     archives,
     >    only the most recent command seems to affect the archives. At the
     end of
     >    these commands, only messages 30,000 to 35,000 are showing up in
     the
     >    archives.

     Are you sure you are not including the --wipe option on the subsequent
     commands? The behavior you describe should not occur unless --wipe is
     specified on the subsequent commands.

     --
     Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
     San Francisco Bay Area, California    better use your sense - B. Dylan
     ------------------------------------------------------
     Mailman-Users mailing list Mailman-Users at python.org
     [1]https://mail.python.org/mailman/listinfo/mailman-users
     Mailman FAQ: [2]http://wiki.list.org/x/AgA3
     Security Policy: [3]http://wiki.list.org/x/QIA9
     Searchable Archives:
     [4]http://www.mail-archive.com/mailman-users%40python.org/
     Unsubscribe:
     [5]https://mail.python.org/mailman/options/mailman-users/awhite%40pdbti.org

   ........................................................................
   Andrew White, PhD
   Associate Director
   DBT-Linehan Board of Certification, Certified DBT Clinician*
   Licensed Clinical Psychologist
   Portland DBT Institute
   (503) 290.3281 (phone)
   (503) 231.8153 (fax)

   Please be aware that e-mail communication can be intercepted in
   transmission or misdirected. This e-mail message and any documents
   attached to it are confidential and may contain information that is
   protected from disclosure by various federal and state laws, including the
   HIPAA privacy rule (45 C.F.R., Part 164). This information is intended to
   be used solely by the entity or individual to whom this message is
   addressed. If you are not the intended recipient, be advised that any use,
   dissemination, forwarding, printing, or copying of this message without
   the sender's written permission is strictly prohibited and may be
   unlawful. Accordingly, if you have received this message in error, please
   notify the sender immediately with a copy to hipaa(at)pdbti.org and
   destroy this message. Please do not include personal identifying
   information such as your birth date, or personal medical information in
   any emails you send to us. No one can diagnose your condition from email
   or other written communications and is not a reliable mechanism for
   emergency communication.

References

   Visible links
   1. https://mail.python.org/mailman/listinfo/mailman-users
   2. http://wiki.list.org/x/AgA3
   3. http://wiki.list.org/x/QIA9
   4. http://www.mail-archive.com/mailman-users@python.org/
   5. https://mail.python.org/mailman/options/mailman-users/awhite@pdbti.org