[Tutor] Inherit from SyntaxError?

[Albert-Jan Roskam]

   On Mar 24, 2023 00:06, Cameron Simpson <cs at cskk.id.au> wrote:

     On 23Mar2023 09:17, Albert-Jan Roskam <sjeik_appie at hotmail.com> wrote:
     >   Sometimes catching MemoryError might come in handy with a
     >   "inside/outside RAM" strategy. In-memory is faster, but not
     scalable.
     >   For instance, something like:
     >   try:
     >       d = {}
     >       for record in records:
     >           d[record.id] = record.something
     >   except MemoryError:
     >       logger.warning("Using slower shelve route")
     >       d = shelve.open("/tmp/lookup.shelve", writeback=True)
     >       atexit.register(d.close)
     >       for record in records:
     >           d[record.id] = record.something
     >   If you know that "records" is always huge, this approach is
     needlessly
     >   complicated. But if it *might* be too big for some machines, it
     might
     >   be nice not having to choose the slow approach, just to be on the
     safe
     >   side.

     Ah. Is the above taken from working code? I've always felt that when a
     MemoryError files things will be comprimised enough that a lot of stuff
     won't subsequently work.

     Taking the above as the example, would you have enough memory to do
     logging or use shelve.open? Even if you explicitly discarded the
     incomplete "d" before trying that (bearing in mind that without invoking
     the garbage collector the memory might not yet be really freed). But
     this opinion is speculation. Have you had joy with the above approach?

   Hi Cameron,
   I've always avoided this approach because I didn't know what behavior I
   could expect. But today I ran a test script and it worked. However,
   ionically, /tmp was quite small on this machine so the huge shelve caused
   a
   ... MemoryError. Oh, and uh, my colleague could no longer ssh to that
   machine. Oops. 😁 In the script below I am basically skipping the shelve
   step for that reason.

   albertjan at mycompany@mymachine:/data/home/albertjan at mycompany$ uname -a

   Linux mymachine.mycompany 4.15.0-207-generic #218-Ubuntu SMP Thu Feb 23
   23:36:05 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

   albertjan at mycompany@mymachine:/data/home/albertjan at mycompany$ python
   --version

   Python 3.7.6

   albertjan at mycompany@mymachine:/data/home/albertjan at mycompany$ cat test.py

   import atexit

   import gc

   import itertools

   import logging

   import shelve

    

   logger = logging.getLogger(__name__)

   logging.basicConfig(format="%(asctime)s: %(message)s", level=logging.INFO)

    

   records = itertools.cycle([1024 * 1024 * "x"])

   try:

       logging.info("started RAM route")

       d = {}

       for id_, record in enumerate(records):

           d[str(id_)] = record[:]

       logging.info("finished RAM route")

   except MemoryError:

       d = {}

       gc.collect()

       d = shelve.open("/tmp/lookup.shelve", writeback=True)

       atexit.register(d.close)

       logging.info("started Shelve route")

       for id_, record in enumerate(records):

           d[str(id_)] = record[:]

           break

       logging.info("finished Shelve route")

   albertjan at mycompany@mymachine:/data/home/albertjan at mycompany$ python
   test.py

   2023-03-24 14:47:05,147: started RAM route

   2023-03-24 16:02:40,816: started Shelve route

   2023-03-24 16:02:40,820: finished Shelve route