[Python-Dev] Investigating Python memory footprint of one real Web application

[INADA Naoki]

On Fri, Jan 20, 2017 at 8:52 PM, Ivan Levkivskyi <levkivskyi at gmail.com> wrote:
> On 20 January 2017 at 11:49, INADA Naoki <songofacandy at gmail.com> wrote:
>>
>> * typing may increase memory footprint, through functions
>> __attributes__ and abc.
>>    * Can we add option to remove or lazy evaluate __attributes__ ?
>
>
> This idea already appeared few times. I proposed to introduce a flag (e.g.
> -OOO) to ignore function and variable annotations in compile.c
> It was decide to postpone this, but maybe we can get back to this idea.
>
> In 3.6, typing is already (quite heavily) optimized for both speed and
> space.
> I remember doing an experiment comparing a memory footprint with and without
> annotations, the difference was few percent.
> Do you have such comparison (with and without annotations) for your app?
> It would be nice to have a realistic number to estimate what would the
> additional optimization flag save.
>
> --
> Ivan
>
>

Hi, Ivan.

I investigated why our app has so many WeakSet today.

We have dozen or hundreds of annotations like Iterable[User] or List[User].
(User is one example of application's domain object.  There are
hundreds of classes).

On the other hand, SQLAlchemy calls isinstance(obj,
collections.Iterable) many times,
in [sqlalchemy.util._collections.to_list](https://github.com/zzzeek/sqlalchemy/blob/master/lib/sqlalchemy/util/_collections.py#L795-L804)
method.

So there are (# of iterable subclasses) weaksets for negative cache,
and each weakset
contains (# of column types) entries.  That's why WeakSet ate much RAM.

It may be slowdown application startup too, because thousands of
__subclasscheck_ is called.

I gave advice to use 'List[User]' instead of List[User] to the team of
the project,
if the team think RAM usage or boot speed is important.

FWIW, stacktrace is like this:

  File "/Users/inada-n/local/py37dbg/lib/python3.7/_weakrefset.py", line 84
    self.data.add(ref(item, self._remove))
  File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 233
    cls._abc_negative_cache.add(subclass)
  File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 226
    if issubclass(subclass, scls):
  File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 226
    if issubclass(subclass, scls):
  File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 191
    return cls.__subclasscheck__(subclass)
  File "venv/lib/python3.7/site-packages/sqlalchemy/util/_collections.py",
line 803
    or not isinstance(x, collections.Iterable):
  File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line 1680
    columns = util.to_list(prop)
  File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line 1575
    prop = self._property_from_column(key, prop)
  File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line 1371
    setparent=True)
  File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line 675
    self._configure_properties()<PasteEnd>

Regards,