From walter at livinglogic.de  Wed May  2 18:05:24 2007
From: walter at livinglogic.de (=?UTF-8?B?V2FsdGVyIETDtnJ3YWxk?=)
Date: Wed, 02 May 2007 18:05:24 +0200
Subject: [Catalog-sig] Two IPython entries
Message-ID: <4638B6C4.4000100@livinglogic.de>

Hello,

Currently there are two entries for IPython in the cheese shop:

http://www.python.org/pypi/IPython (for 0.5.0)
http://www.python.org/pypi/ipython (for 0.8.0)

However we can't delete the old entry for 0.5.0 because it belongs to
the user rodholland. (See this thread on the ipython-dev mailing list
http://lists.ipython.scipy.org/pipermail/ipython-dev/2007-May/002893.html)

Could one of the cheese shop maintainers delete the old package?

Servus,
   Walter

From amk at amk.ca  Thu May  3 20:08:04 2007
From: amk at amk.ca (A.M. Kuchling)
Date: Thu, 3 May 2007 14:08:04 -0400
Subject: [Catalog-sig] Untested patch for PyPI exceptions
Message-ID: <20070503180804.GA15318@localhost.localdomain>

The completely untested patch below should remedy the occasional
exceptions that PyPI is occasionally raising.  Can someone provide a
sanity check for this patch?  (I'm not going to deploy it -- not
familiar enough with the setup on the live site.)

--amk

Index: flamenco.py
===================================================================
--- flamenco.py	(revision 452)
+++ flamenco.py	(working copy)
@@ -1,6 +1,7 @@
 #!/usr/bin/python
 
 import sys, pprint, cgi, urllib, os, time, cStringIO
+import psycopg
 
 def flatten(d):
     l = d[1]
@@ -114,7 +115,8 @@
             # overlapping update, just release the lock
             self.store.commit()
             return
-        self.cursor.execute("delete from browse_tally")
+        try:
+            self.cursor.execute("delete from browse_tally")
         # Using PostgreSQL COPY here, instead of a series of INSERT statements
         s = cStringIO.StringIO()
         for _, _, items in self.list_choices()[1]:
@@ -123,6 +125,8 @@
         s.seek(0)
         self.cursor.copy_from(s, 'browse_tally')
         self.cursor.execute("update timestamps set value=now() where name='browse_tally'")
+        except psycopg.ProgrammingError:
+            pass
         self.store.commit()
 
     def get_matches(self, addl_fields=[]):

From exarkun at divmod.com  Thu May  3 20:53:14 2007
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Thu, 3 May 2007 14:53:14 -0400
Subject: [Catalog-sig] Untested patch for PyPI exceptions
In-Reply-To: <20070503180804.GA15318@localhost.localdomain>
Message-ID: <20070503185314.19381.1871284319.divmod.quotient.7998@ohm>

On Thu, 3 May 2007 14:08:04 -0400, "A.M. Kuchling" <amk at amk.ca> wrote:
>The completely untested patch below should remedy the occasional
>exceptions that PyPI is occasionally raising.  Can someone provide a
>sanity check for this patch?  (I'm not going to deploy it -- not
>familiar enough with the setup on the live site.)
>

The indentation looks wrong, at least.

Jean-Paul

From amk at amk.ca  Thu May  3 21:22:37 2007
From: amk at amk.ca (A.M. Kuchling)
Date: Thu, 3 May 2007 15:22:37 -0400
Subject: [Catalog-sig] Untested patch for PyPI exceptions
In-Reply-To: <20070503185314.19381.1871284319.divmod.quotient.7998@ohm>
References: <20070503180804.GA15318@localhost.localdomain>
	<20070503185314.19381.1871284319.divmod.quotient.7998@ohm>
Message-ID: <20070503192237.GB16649@localhost.localdomain>

On Thu, May 03, 2007 at 02:53:14PM -0400, Jean-Paul Calderone wrote:
> The indentation looks wrong, at least.

Oops, thanks!  Attempt #2...

--amk

Index: flamenco.py
===================================================================
--- flamenco.py	(revision 452)
+++ flamenco.py	(working copy)
@@ -1,6 +1,7 @@
 #!/usr/bin/python
 
 import sys, pprint, cgi, urllib, os, time, cStringIO
+import psycopg
 
 def flatten(d):
     l = d[1]
@@ -114,15 +115,18 @@
             # overlapping update, just release the lock
             self.store.commit()
             return
-        self.cursor.execute("delete from browse_tally")
-        # Using PostgreSQL COPY here, instead of a series of INSERT statements
-        s = cStringIO.StringIO()
-        for _, _, items in self.list_choices()[1]:
-            for path, packages in items:
-                s.write('%d\t%d\n' % (self.trove.getid(path), len(packages)))
-        s.seek(0)
-        self.cursor.copy_from(s, 'browse_tally')
-        self.cursor.execute("update timestamps set value=now() where name='browse_tally'")
+        try:
+            self.cursor.execute("delete from browse_tally")
+            # Using PostgreSQL COPY here, instead of a series of INSERT statements
+            s = cStringIO.StringIO()
+            for _, _, items in self.list_choices()[1]:
+                for path, packages in items:
+                    s.write('%d\t%d\n' % (self.trove.getid(path), len(packages)))
+            s.seek(0)
+            self.cursor.copy_from(s, 'browse_tally')
+            self.cursor.execute("update timestamps set value=now() where name='browse_tally'")
+        except psycopg.ProgrammingError:
+            pass
         self.store.commit()
 
     def get_matches(self, addl_fields=[]):

From amk at amk.ca  Fri May  4 13:44:46 2007
From: amk at amk.ca (A.M. Kuchling)
Date: Fri, 4 May 2007 07:44:46 -0400
Subject: [Catalog-sig] Fwd: bug in log out link?
Message-ID: <20070504114446.GA10671@localhost.localdomain>

Forwarded.

--amk
-------------- next part --------------
An embedded message was scrubbed...
From: Ian Zimmerman <itz at madbat.mine.nu>
Subject: Re: Complete your Cheese Shop registration
Date: 03 May 2007 23:42:30 -0400
Size: 3233
Url: http://mail.python.org/pipermail/catalog-sig/attachments/20070504/4f464f49/attachment.mht 

From martin at v.loewis.de  Sun May  6 10:35:52 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 06 May 2007 10:35:52 +0200
Subject: [Catalog-sig] Untested patch for PyPI exceptions
In-Reply-To: <20070503180804.GA15318@localhost.localdomain>
References: <20070503180804.GA15318@localhost.localdomain>
Message-ID: <463D9368.2020903@v.loewis.de>

A.M. Kuchling schrieb:
> The completely untested patch below should remedy the occasional
> exceptions that PyPI is occasionally raising.  Can someone provide a
> sanity check for this patch?  (I'm not going to deploy it -- not
> familiar enough with the setup on the live site.)

I don't understand why this is necessary (although I trust that these
exceptions really occur).

Can any expert on Postgres please shed a light on it? The code currently
reads

        self.cursor.execute("lock table browse_tally")
        if self.is_cache_current():
            # overlapping update, just release the lock
            self.store.commit()
            return
        self.cursor.execute("delete from browse_tally")
        ...
        self.cursor.execute("update timestamps set value=now() where
name='browse_tally'")
        self.store.commit()

Now, this code gets an exception

psycopg.ProgrammingError: ERROR:  could not serialize access due to
concurrent update

delete from browse_tally

How is that possible? This code is the only code code updating
browse_tally (although multiple simultaneous processes may run the
same code); I would expect that the "lock table" statement prevents
concurrent updates.

Any help appreciated.

Regards,
Martin

From amk at amk.ca  Tue May  8 14:30:20 2007
From: amk at amk.ca (A.M. Kuchling)
Date: Tue, 8 May 2007 08:30:20 -0400
Subject: [Catalog-sig] Fwd: [PyPI] Internal Error
Message-ID: <20070508123020.GA7006@localhost.localdomain>

Here's a copy of the sporadically-received tracebacks from PyPI.

--amk

-------------- next part --------------
An embedded message was scrubbed...
From: richard at python.org
Subject: [PyPI] Internal Error
Date: Tue,  8 May 2007 14:17:40 +0200 (CEST)
Size: 3000
Url: http://mail.python.org/pipermail/catalog-sig/attachments/20070508/7b860ecd/attachment.mht 

From martin at v.loewis.de  Fri May 11 09:54:34 2007
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 11 May 2007 09:54:34 +0200
Subject: [Catalog-sig] Fwd: [PyPI] Internal Error
In-Reply-To: <20070508123020.GA7006@localhost.localdomain>
References: <20070508123020.GA7006@localhost.localdomain>
Message-ID: <4644213A.8070001@v.loewis.de>

A.M. Kuchling schrieb:
> Here's a copy of the sporadically-received tracebacks from PyPI.

I think I have now fixed this problem, at least in the common case.
The problem was that I had a time-stamp associated with the browse
tally, to regenerate the tally if it is too old. I read the time
stamp, lock the tally table, regenerate it, and update the time stamp.

Now, if two processes do the same simultaneously, they first acquire
a read lock for time stamp (implicitly as a side effect of the
select). Then they ask for a table lock, and one of them gets it
and performs the write. That one then writes the time-stamp,
requiring a write lock for the time-stamp, and attempts to upgrade
the read lock. For this, the other reader would need to go away,
but it waits for a write lock that we hold - a deadlock occurs.

I'm not quite sure how Postgres proceeds exactly. It somehow
breaks the locks, apparently breaking the read lock, allowing
the upgrade to the write lock. So the entire update succeeds,
but the second writer fails.

The solution is to commit the read operation before locking the
table, to release the read lock. In the new transaction, only
a table lock is required at first, and no deadlock arises.

I have changed PyPI to operate that way, and on my local
installation, the error went away.

I'm unsure whether there could be conflicts with other
write operations, but at the moment, I can't see how.

Regards,
Martin