Is there a more efficient threading lock?

Skip Montanaro skip.montanaro at gmail.com
Sun Feb 26 12:53:45 EST 2023


Thanks for the various replies. The program originally started out
single-threaded. I wandered down the multi-threaded path to see if I could
get a performance boost using Sam Gross's NoGIL fork
<https://github.com/colesbury/nogil-3.12>. I was pretty sure the GIL would
limit multi-threading performance on a stock Python interpreter. When I
first switched to threads, I didn't have a lock around the one or two
places which called out to the TextBlob <https://textblob.readthedocs.io/>/NLTK
stuff. The use of threading.Lock was the obvious simplest choice, and it
solved the crash I saw without it. I'm still thinking about using queues to
communicate between the email processing threads and the TextBlob & SQLite
processing stuff.

I had been doing a bit of pre- and post-processing of the default TextBlob
noun phrase generation, but I wasn't happy with it, so I decided to
experiment with an alternate noun phrase extractor
<https://textblob.readthedocs.io/en/dev/api_reference.html?highlight=ConllExtractor#textblob.en.np_extractors.ConllExtractor>.
I was happier with that, so ripped out most of the ad hoc stuff I was
doing. While doing this code surgery, I moved back to 3.11 to have a more
trusty Python interpreter. (I've yet to encounter a problem with NoGIL,
just cutting back on moving parts, and wasn't seeing any obvious
performance gains.)

As for SQLite and multi-threading, I figured if the core devs hadn't yet
gotten around to making it available then it probably wasn't
straightforward. I wasn't willing to tackle that.

So, I'll keep messing around. It's all just for fun
<https://www.smontanaro.net/CR> anyway.

Skip


More information about the Python-list mailing list