[ZODB] Multithreaded connection pooling support for ZODB databases?

[Etienne Robillard]

Hi,

I would like to know if you could please share some input about this 
ThreadConnectionPool class for libschevo/ZODB. :)


Thanks!
Etienne

Le 2018-04-20 à 04:49, Etienne Robillard a écrit :
> Heads up people!
>
> I've finally managed to make a working `ThreadedConnectionPool` class 
> for supporting multiple ZODB databases connections on top of libschevo 
> API! :)
>
> Technically, I decided to use gevent.Greenlet under the hood to spawn 
> multiple cooperative ClientStorage client/server connections.
>
> I'm also making all my testing and development under PyPy 5.9.
>
> You can check out the code here: 
> https://bitbucket.org/tkadm30/libschevo/commits/37feb029615d76f3d81233990e509b6eb6ffb5d7
>
> Comments are welcome! :)
>
> Etienne
>
>
> Le 2018-04-16 à 18:27, Etienne Robillard a écrit :
>> Hi Jason,
>>
>> I just made some more changes to my ThreadedConnectionPool class 
>> here: 
>> https://bitbucket.org/tkadm30/django-hotsauce/commits/81d2e8f30019840d9c8bbdf7f82df6de2be024fc
>>
>> In summary, the `ThreadedConnectionPool` class is now a subclass of 
>> `threading.Thread` and is extending the `run` method to populate a 
>> thread local dictionary with ClientStorage connections.
>>
>> Kind regards,
>>
>> Etienne
>> Le 2018-04-16 à 09:14, Jason Madden a écrit :
>>>>>> On Apr 15, 2018, at 8:17 PM, Etienne Robillard 
>>>>>> <tkadm30 at yandex.com> wrote:
>>>>>>
>>>>>> I would like to define a `ThreadedConnectionPoll` class to allow 
>>>>>> multithreaded caching of ZODB databases into memory.
>>>
>>> The ZODB.DB object is *already* thread safe for connection 
>>> management. You shouldn't need to do anything other than use 
>>> ZODB.DB().open() and conn.close() from all the threads you're using. 
>>> (That is, create exactly one ZODB.DB object for each database you 
>>> are using and share that object amongst your threads, using 
>>> db.open()/conn.close() as needed. For storages like RelStorage that 
>>> have caches at the ZODB.DB level---which are also thread safe---this 
>>> is important.)
>>>
>>> If you are managing multiple databases and want to be able to make 
>>> references between them (a multiple-database) it is critical that 
>>> they share the same `databases` object. But that is an advanced usage.
>>>
>>>
>>>> Here's a updated version of my code so far using threading.local :
>>>>
>>>> from threading import local
>>>> from notmm.dbapi.orm import ClientStorageProxy
>>>>
>>>> class ThreadedConnectionPool(object):
>>>>
>>>>      def __init__(self, d, debug=True):
>>>>          if debug:
>>>>              assert isinstance(d, dict) == True
>>>>          local.pool = d
>>>>          for key,value in local.pool.iteritems():
>>> Unless you left out part of the code, it appears that you're trying 
>>> to set (and read) a class attribute on the threading.local class. 
>>> This does not create thread-local state. It is also not portable and 
>>> doesn't work when local is an extension type:
>>>
>>> py> from threading import local
>>> py> import sys
>>> py> sys.version_info # CPython 3.7; same result in 2.7
>>> sys.version_info(major=3, minor=7, micro=0, releaselevel='beta', 
>>> serial=3)
>>> py> local
>>> <class '_thread._local'>
>>> py> local.foo = 1
>>> Traceback (most recent call last):
>>>    File "<stdin>", line 1, in <module>
>>> TypeError: can't set attributes of built-in/extension type 
>>> '_thread._local'
>>>
>>> pypy> sys.version_info # pypy 2
>>> (major=2, minor=7, micro=13, releaselevel='final', serial=42)
>>> pypy> from threading import local
>>> pypy> local.foo = 1
>>> Traceback (most recent call last):
>>>    File "<stdin>", line 1, in <module>
>>> TypeError: can't set attributes on type object '_local'
>>>
>>> It generally also shouldn't be necessary as the parameters to 
>>> __init__ are preserved and used again in each thread; you can use 
>>> this to share information if those parameters are mutable:
>>>
>>> py> from threading import get_ident
>>> py> from threading import Thread
>>> py> class MyLocal(local):
>>> ...     def __init__(self, l):
>>> ...         print("Creating in thread", get_ident(), "param", l)
>>> ...         l.append(get_ident())
>>> ...
>>> py> l = []
>>> py> mylocal = MyLocal(l)
>>> Creating in thread 140736147411840 param []
>>> py> l
>>> [140736147411840]
>>> py> def target():
>>> ...     mylocal.foo = 1
>>> ...
>>> py> t = Thread(target=target)
>>> py> t.start()
>>> Creating in thread 123145538318336 param [140736147411840]
>>> None
>>> py> l
>>> [140736147411840, 123145538318336]
>>>
>>> Jason
>>>
>>
>

-- 
Etienne Robillard
tkadm30 at yandex.com
https://www.isotopesoftware.ca/