[Async-sig] [python-tulip] asyncio.Lock equivalent for multiple processes

[Ludovic Gasc]

Hi Nickolai,

Thanks for your suggestions, especially for the file system lock: We don't
have often locks, but we must be sure it's locked.

For 1) and 4) suggestions, in fact we have several systems to sync and also
a PostgreSQL transaction, the request must be treated by the same worker
from beginning to end and the other systems aren't idempotent at all, it's
"old-school" proprietary systems, good luck to change that ;-)

Regards.
--
Ludovic Gasc (GMLudo)

2018-04-17 12:46 GMT+02:00 Nickolai Novik <nickolainovik at gmail.com>:

> Hi, redis lock has own limitations and depending on your use case it may
> or may not be suitable [1]. If possible I would redefine problem and also
> considered:
> 1) create worker per specific resource type to avoid locking
> 2) optimistic locking
> 3) File system lock like in twisted, but not sure about performance and
> edge cases there
> 4) make operation on resource idempotent
>
> [1] http://martin.kleppmann.com/2016/02/08/how-to-do-
> distributed-locking.html
> [2] https://github.com/twisted/twisted/blob/e38cc25a67747899c6984d6ebaa8d3
> d134799415/src/twisted/python/lockfile.py
>
> On Tue, 17 Apr 2018 at 13:01 Ludovic Gasc <gmludo at gmail.com> wrote:
>
>> Hi Roberto,
>>
>> Thanks for the pointer, it's exactly the type of feedbacks I'm looking
>> for: Ideas that are out-of-box of my confort zone.
>> However, in our use case, we are using gunicorn, that uses forks instead
>> of multiprocessing to my knowledge, I can't use multiprocessing without to
>> remove gunicorn.
>>
>> If somebody is using aioredlock in his project, I'm interested by
>> feedbacks.
>>
>> Have a nice week.
>>
>>
>> --
>> Ludovic Gasc (GMLudo)
>>
>> 2018-04-17 7:19 GMT+02:00 Roberto Martínez <robertomartinezp at gmail.com>:
>>
>>>
>>> Hi,
>>>
>>> I don't know if there is a third party solution for this.
>>>
>>> I think the closest you can get today using the standard library is
>>> using a multiprocessing.manager().Lock (which can be shared among
>>> processes) and call the lock.acquire() function with
>>> asyncio.run_in_executor(), using a ThreadedPoolExecutor to avoid blocking
>>> the asyncio event loop.
>>>
>>> Best regards,
>>> Roberto
>>>
>>>
>>> El mar., 17 abr. 2018 a las 0:05, Ludovic Gasc (<gmludo at gmail.com>)
>>> escribió:
>>>
>>>> Hi,
>>>>
>>>> I'm looking for a equivalent of asyncio.Lock (
>>>> https://docs.python.org/3/library/asyncio-sync.html#asyncio.Lock) but
>>>> shared between several processes on the same server, because I'm migrating
>>>> a daemon from mono-worker to multi-worker pattern.
>>>>
>>>> For now, the closest solution in term of API seems aioredlock:
>>>> https://github.com/joanvila/aioredlock#aioredlock
>>>> But I'm not a big fan to use polling nor with a timeout because the
>>>> lock I need is very critical, I prefer to block the code than unlock with
>>>> timeout.
>>>>
>>>> Do I miss a new awesome library or do you have an easier approach ?
>>>>
>>>> Thanks for your responses.
>>>> --
>>>> Ludovic Gasc (GMLudo)
>>>>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20180417/927aa81e/attachment-0001.html>