[pypy-dev] funding/popularity?

[Paolo Giarrusso]

On Thu, Dec 23, 2010 at 12:14, Armin Rigo <arigo at tunes.org> wrote:
> Hi Dima,
>
> On Wed, Dec 22, 2010 at 11:21 PM, Dima Tisnek <dimaqq at gmail.com> wrote:
>> --- Comment #4 from Boris Zbarsky (:bz) <bzbarsky at mit.edu> 2010-12-22
>> 13:43:23 PST ---
>> So what I see this page do, in horizontal mode, is create 17 canvases each of
>> which is width="816" height="3587".  That means that each of them has a backing
>> store of 816*3587*4 = 11,707,968 bytes.  So that's about 200MB of memory usage
>> right there.
>>
>> I have no idea why they feel a need for 17 huge canvases, but if they want
>> them, that's how much memory they'll take...
>
> That looks very similar to an issue with PyPy's own GC, in which
> ctypes.create_string_buffer() returns objects which tend to be GC'ed
> late.  That's because the string buffer object in ctypes appears (to
> PyPy's GC) to be just a small object, even though it actually
> references a potentially large piece of raw memory.

Bigger objects are not collected sooner by typical GC algorithms* -
they might just fill the heap more quickly and trigger earlier a GC
invocation.
>From your description, I guess that what you describe as "raw memory"
is not accounted in the stats triggering GC. That would explain the
behavior you describe, and suggest an easy fix. Indeed, one could wrap
the raw-memory allocator to create and update such stats; then the
heap-overflow check could consider them to decide whether to trigger
GC. The details of the modified heap-overflow check would probably not
be entirely trivial, but still doable.

* "Typical" GC algorithms here includes copying, mark-sweep, and
mark-compact collectors, including generational variants; even
collectors having a large object space (typically reclaimed through
mark-sweep) should still typically collect small objects often (by
using the heuristic described above to trigger GC).

Best regards

> Similarly, my
> vague guess about the above is that the 17*11MB of memory are hold by
> 17 small objects which firefox's GC think don't need to be collected
> particularly aggressively.

-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/