[ZODB] Multithreaded connection pooling support for ZODB databases?

[Etienne Robillard]

Heads up people!

I've finally managed to make a working `ThreadedConnectionPool` class 
for supporting multiple ZODB databases connections on top of libschevo 
API! :)

Technically, I decided to use gevent.Greenlet under the hood to spawn 
multiple cooperative ClientStorage client/server connections.

I'm also making all my testing and development under PyPy 5.9.

You can check out the code here: 
https://bitbucket.org/tkadm30/libschevo/commits/37feb029615d76f3d81233990e509b6eb6ffb5d7

Comments are welcome! :)

Etienne


Le 2018-04-16 à 18:27, Etienne Robillard a écrit :
> Hi Jason,
>
> I just made some more changes to my ThreadedConnectionPool class here: 
> https://bitbucket.org/tkadm30/django-hotsauce/commits/81d2e8f30019840d9c8bbdf7f82df6de2be024fc
>
> In summary, the `ThreadedConnectionPool` class is now a subclass of 
> `threading.Thread` and is extending the `run` method to populate a 
> thread local dictionary with ClientStorage connections.
>
> Kind regards,
>
> Etienne
> Le 2018-04-16 à 09:14, Jason Madden a écrit :
>>>>> On Apr 15, 2018, at 8:17 PM, Etienne Robillard 
>>>>> <tkadm30 at yandex.com> wrote:
>>>>>
>>>>> I would like to define a `ThreadedConnectionPoll` class to allow 
>>>>> multithreaded caching of ZODB databases into memory.
>>
>> The ZODB.DB object is *already* thread safe for connection 
>> management. You shouldn't need to do anything other than use 
>> ZODB.DB().open() and conn.close() from all the threads you're using. 
>> (That is, create exactly one ZODB.DB object for each database you are 
>> using and share that object amongst your threads, using 
>> db.open()/conn.close() as needed. For storages like RelStorage that 
>> have caches at the ZODB.DB level---which are also thread safe---this 
>> is important.)
>>
>> If you are managing multiple databases and want to be able to make 
>> references between them (a multiple-database) it is critical that 
>> they share the same `databases` object. But that is an advanced usage.
>>
>>
>>> Here's a updated version of my code so far using threading.local :
>>>
>>> from threading import local
>>> from notmm.dbapi.orm import ClientStorageProxy
>>>
>>> class ThreadedConnectionPool(object):
>>>
>>>      def __init__(self, d, debug=True):
>>>          if debug:
>>>              assert isinstance(d, dict) == True
>>>          local.pool = d
>>>          for key,value in local.pool.iteritems():
>> Unless you left out part of the code, it appears that you're trying 
>> to set (and read) a class attribute on the threading.local class. 
>> This does not create thread-local state. It is also not portable and 
>> doesn't work when local is an extension type:
>>
>> py> from threading import local
>> py> import sys
>> py> sys.version_info # CPython 3.7; same result in 2.7
>> sys.version_info(major=3, minor=7, micro=0, releaselevel='beta', 
>> serial=3)
>> py> local
>> <class '_thread._local'>
>> py> local.foo = 1
>> Traceback (most recent call last):
>>    File "<stdin>", line 1, in <module>
>> TypeError: can't set attributes of built-in/extension type 
>> '_thread._local'
>>
>> pypy> sys.version_info # pypy 2
>> (major=2, minor=7, micro=13, releaselevel='final', serial=42)
>> pypy> from threading import local
>> pypy> local.foo = 1
>> Traceback (most recent call last):
>>    File "<stdin>", line 1, in <module>
>> TypeError: can't set attributes on type object '_local'
>>
>> It generally also shouldn't be necessary as the parameters to 
>> __init__ are preserved and used again in each thread; you can use 
>> this to share information if those parameters are mutable:
>>
>> py> from threading import get_ident
>> py> from threading import Thread
>> py> class MyLocal(local):
>> ...     def __init__(self, l):
>> ...         print("Creating in thread", get_ident(), "param", l)
>> ...         l.append(get_ident())
>> ...
>> py> l = []
>> py> mylocal = MyLocal(l)
>> Creating in thread 140736147411840 param []
>> py> l
>> [140736147411840]
>> py> def target():
>> ...     mylocal.foo = 1
>> ...
>> py> t = Thread(target=target)
>> py> t.start()
>> Creating in thread 123145538318336 param [140736147411840]
>> None
>> py> l
>> [140736147411840, 123145538318336]
>>
>> Jason
>>
>

-- 
Etienne Robillard
tkadm30 at yandex.com
https://www.isotopesoftware.ca/