[Python-ideas] More details in MemoryError

[Charles-François Natali]

2013/1/21 Oleg Broytman <phd at phdru.name>:
>    I'd very much like to see a situation when a program can survive
> MemoryError.

Let's say your using an image processing program.
You have several images open on which you've been working for a couple
minutes/hours.
You open a new one, and it's so large that it results in MemoryError :
instead of just losing all your current work (yeah, the program should
support auto-save anyway, but let's pretend it doesn't), the program
catches MemoryError, and displays a popup saying "No enough memory to
process this image".

Now, sure, there are cases where an OOM condition will result in
thrashing to death, or simply because of overcommit malloc() will
never return NULL and you'll get nuked by the OOM killer, but
depending on your operating system and allocation pattern, there are
times when you can reasonably recover from a MemoryError.
Also, a memory allocation failure doesn't necessarily mean you're OOM,
it could be that youve exhausted your address space (on 32-bit), or
hit RLIMIT_VM/RLIMIT_DATA.

2013/1/21 Benjamin Peterson <benjamin at python.org>:
> What is this useful for?

Even if the exception isn't caught, if the extra information gets
dumped in the traceback, it can be used for post-mortem debugging (to
help distinguish between OOM, address space exhaustion, heap
fragmentation, overflow in computation of malloc() argument, etc).


So I think it could probably be useful, but I see two problems:
- right now, the amount of memory isn't tracked. IIRC, Antoine added
recently a counter for allocated blocks, not bytes
- the exception is raised at the calling site where the allocation
routine failed (this comes from Modules/_pickle.c):
"""
    PyMemoTable *memo = PyMem_MALLOC(sizeof(PyMemoTable));
        if (memo == NULL) {
        PyErr_NoMemory();
        return NULL;
    }
"""

So we can't easily capture the current allocated memory and the
requested memory (the former could probably be retrieved in
PyErr_NoMemory(), but the later would require modifying every call
site and repeating it).