changing dicts in a threaded environment ?

[Bart Thate]

Terry !

thnx for replying, feedback is the thing i need ;] sorry for the late
reply, after your mail i have been coding like hurt (so many hours nonstop
hacking hurts in the chair). As i started with all the code in 1 file (what
i get when i think something is cool), i splitted the stuff up into proper
modules, so i can show that to explain what i am up to.

I come to this code a long time ago, during the programming of my previous
bots, where co coder maze introduced me to what he called a LazyDict. Dict
with dotted access for attributes instead of __getitem__, i like how the
code looks when dotted acces is used. While this contruct found its way
into my code, i had to deal with something different and this is "how to
adapt my data - storage - to changes in the code" problem.

All the databases i tried used some sort of schema to define the data and
onces that needs to change, reflection needed to take place to update the
storage of data. I need to be able to add attributes to objects without no
need to change or reflect anything. So i took the LazyDict and added json
dump and load functionality to it. This worked great in terms of
flexibility in the code, but load on it is very low, so not to be compared
with the scale etc that normal databases have to deal with. Soon i want to
add a .send(JID) to it so i can send a object via JSON to the other side
and have it reconstruct there. Kinda like Pyro based on JSON, been
experimenting with that with Aim during our endless "let's code a new
collective" attempts, where the "send the pickle over the wire" stuff was
not secure, the use of XML to make it secure made it a true bandwidth hog,
and parsing JSON is still considered secure, because no eval gets used when
deconstructing.

In first i am not interested in deconstructing objects from the JSON yet,
my first intention is to make a JSON dict replication system, where i can
put data in a dict, put JSON of that dict over the wire to various other
pieces of my software and have it possible to authenticate the correctness
of the data. This is because i want - eventually - to have my code run in a
device where the data it generates can be used as proof in a court of law.
This is to make some modern version of a diary, where one can register all
things happening during a period of time.

So i have to make this  1 LazyDict, JSON saveable and sendable etc, objects
thats is going to be the core of what i doing and i need to properly design
it and todo that properly i need feedback on it ;]

Thank for your feedback Terry, it is clear to me now that i need to do the
locking myself, thing is i need to have this object behave excactly as
normal standard object (dict in this case) so i want to know if there are
any pointers to docs that discuss the semantics of dicts esp. what needs to
be locked properly to ensure that there is only one accessor at the time.

Question to all of you is something like "if you want 1 basic type added to
python 4, what would it's semantics be ?"

See https://github.com/feedbackflow/life for the code i am working on now ;]

https://github.com/feedbackflow/life/blob/master/life/__init__.py - for the
big O class that is to become the core of it all, pun intended ;]

Looking forward to hear what you and all the others reading this are
thinking.

Bart, thinking of "what if i stored the JSON in git ?" and "would mercurial
then not be better" haha


p.s. code is not in a "good shape" yet, it is just that i would take too
long for me to respond and 20 hours work days are killing me to get this
right "the first time" ;]
p.s.s trbs your code is in the moment plugs are going to be added, need the
core solid first ;]


On Tue, Nov 27, 2012 at 8:18 PM, Terry Reedy <tjreedy at udel.edu> wrote:

> On 11/27/2012 7:53 AM, Bart Thate wrote:
>
> [Answers based on reading without thread experience.]
>
>
>  i use python3 now and i need to be able to remove elements from a dict
>> in a thread safe manner.
>>
>
> Essentially no change from py2.
>
>
>  kinda like a Queue.Queue thing but then in a dict, so i can pass arond
>>   my dict based objects as parameters arond without having to wonder if
>> it gets properly locked.
>>
>
> As I understand it, dicts do not get locked unless you do it.
>
>  So not only do i need to get this solved:
>>
>> Issue #14417 <http://bugs.python.org/14417>**: Mutating a dict during
>>
>> lookup now restarts the lookup instead of raising a RuntimeError (undoes
>> issue #14205 <http://bugs.python.org/14205>**).
>>
>
> As I understand #14417, you should be explicitly locking dicts. The issue
> in #14205 was that people doing mutations in just one thread and lookups in
> others usually got away without locking becuase of recursive retries, but
> occasionally crashed the interpreter because of them. The first fix was to
> prevent crashes by removing retries. But that would break programs that
> naively worked because of them. The second fix was to do iterative retries
> instead, so a thread might occasionally hang, but without crashing.
>
> As I understand it, it is better to not be a naive user.
>
>
>  i also need to lock all the actual modifying underlying "real" stuff as
>> well not just the iterator or view or whatever i don't know yet ;]
>>
>
> I am not sure what you are asking.
>
>
>> So my question is kinda like, is a dict possible in the same way the
>> Queue.Queue is now made truely thread safe ?
>>
>
> A custom dict might be but the builtin is not. Queue is somewhat unique as
> a builtin designed for threads.
>
>
>  Also is there something like a select.select for queues ?
>>
>
> Have you searched?
>
>
>  I want to pass a dict to a thead and then have a watcher on the dicts
>> state if result have arrived.
>>
>
> Builtin dicts do not have an 'I have changed flag. You would need to
> subclass or probably better, wrap a dict using the appropriate pattern.
> Perhaps done already, but you want a wrapper that does both locking and
> watcher notification.
>
> --
> Terry Jan Reedy
>
> --
> http://mail.python.org/**mailman/listinfo/python-list<http://mail.python.org/mailman/listinfo/python-list>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20121129/f08c4bb9/attachment.html>