Access violation when no memory

[Phil Mayes]

Vladimir Marangozov wrote in message <395222DD.353DDD8F at inrialpes.fr>...
>Phil Mayes wrote:
>>
>> No, I understand that - my program is explicitly designed to consume
>> all memory by progressively consuming smaller and smaller blocks,
>> throwing exceptions at each size.
>
>Whatever your task is, I strongly doubt that consuming all the memory
>of your machine is really what you want to do. Not from Python, at least.

No, of course not; I have a 15,000-line program in beta that crashes on
several users' machines, and the sample I gave is the distillation of
weeks of painful debugging.

>The net effect of your program is zeroing the memory the OS is kind enough
>to allocate for Python. It's not really Python's fault in such extreme
>situations. Tim Peters reported (and one could try it easily with a
>malloc(BIG) call) that Linux overbooks it's memory, that is, the malloc
>succeeds, but accessing everything subsequently segfaults because the
>requested mem size is not physically present (RAM + swap included).
>
>So this boils down on how the OS handles such extreme cases. Python
>usually does not trap OS signals, etc. and it cannot handle OS-level
>mem overflows. All it does is trusting C malloc and checking the return
>values for success or failure.

Yes, a page-fault on a committed malloc(BIG) is an OS issue, but that's
not what is happening, which is:
  a) a MemoryError exception is thrown
  b) handling the MemoryError involves getting a traceback object
  c) there's no memory for it, so NULL is pushed
  d) the pop uses its address

>> So I need a ceval.c expert to vet my change.
>
>Forget about it <wink>. Nobody is convinced that checking for NULL the
>top of stack after popping its value would solve any problem. Your example
>crashes on my combo in the builtin dict methods (& elsewhere).

Yes, I am concerned that I have not found all the places that Python
can fail under low-memory conditions.

> This is
>really low-level OS stuff, and Python has not builtin emergency procedures
>for these cases. I don't know why you want to eat all your memory, but
>I'd suggest changing the approach or write a C module that deals with
>the memory issues you're looking for.

In my experience, error-handling is one of the hardest tasks in software
development, and ends up being about 20% of the development effort.
If Python is to be used for widely-distributed software, it has to be
robust and handle such extreme cases gracefully, and I suspect that the
uses to which it has traditionally been put have not fully exercised
some error situations.

>--
>       Vladimir MARANGOZOV          | Vladimir.Marangozov at inrialpes.fr
>http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252

--
Phil Mayes    pmayes AT olivebr DOT com