Help with large data set and fatal python error (pysqlite)

[Vineet Jain]

> or pyrex?  If the latter, it could be a bug in the wrapper generator.
> If not, it might help to post some of your code which acts on the
> PyObjects.

I've been able to recreate the problem with the following code segment. It
seems that the problem comes when you have multple db files open at the same
time. In my application I have many database files open at the same time.
I'm not using threads so all access to the db files are done sequenetially.

At the end of the program I get the following error:

Fatal Python error: deallocating None

This application has requested the Runtime to terminate it in an unusual
way.
Please contact the application's support team for more information.

If I change the numInnerLoop to 5, 10, 20, or 30 it works however anything
over 35 gives the above error. In my actual application, when run for a
larger set, the error can come in the middle of the program and is causing
me a lot of grief.

Any help would be really appreciated.




-------------------Problem Code----------------------------
import sqlite
import gc
import sys, os

#initialize database

dbFileName      = "c:/test.db"
numRecords      = 1000

os.remove(dbFileName)
conn    = sqlite.connect(database=dbFileName,autocommit=0)
cur     = conn.cursor()
cur.execute("create table t (field1, field2, field3, field4, field5, field6,
field7)")
for i in range(numRecords):
    cur.execute("insert into t values(1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0)")
conn.commit()
conn.close()

numInnerLoop = 50
conn         = [None]*numInnerLoop
cur          = [None]*numInnerLoop
tempStore    = [None]*numInnerLoop
qry          = "select * from t"

for k in range(1):
    for i in range(numInnerLoop):
        conn[i]   = sqlite.connect(database=dbFileName,autocommit=1)
        cur[i]    = conn[i].cursor()

        sys.stdout.write('OuterLoop: %d   Inner Loop :  %d\r' % (k, i))

        tempStore[i] = []
        cur[i].execute(qry)
        dbRowList = cur[i].fetchall()
        for j in range(numRecords):
            for l in range(6):
                tempStore[i].append(float(dbRowList[j][l]))

        dbRowList = None
        gc.collect()

for i in range(numInnerLoop):
    conn[i].close()


On the other hand when I work with only one database connection. Things work
fine:

--------------------------Ok
Code----------------------------------------------------
import sqlite
import gc
import sys, os

#initialize database

tempStore       = [None]*400
dbFileName      = "c:/test.db"


os.remove(dbFileName)
conn    = sqlite.connect(database=dbFileName,autocommit=0)
cur     = conn.cursor()
cur.execute("create table t (field1, field2, field3, field4, field5, field6,
field7)")
for i in range(10000):
    cur.execute("insert into t values(1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0)")
conn.commit()
conn.close()

conn    = sqlite.connect(database=dbFileName,autocommit=1)
cur     = conn.cursor()

qry     = "select * from t"
for k in range(1):
    for i in range(300):

        sys.stdout.write('OuterLoop: %d   Inner Loop :  %d\r' % (k, i))

        tempStore[i] = []
        cur.execute(qry)
        dbRowList = cur.fetchall()
        for j in range(10000):
            for l in range(6):
                tempStore[i].append(float(dbRowList[j][l]))

        dbRowList = None
        gc.collect()

conn.close()