Size limit on compiling?

[Bengt Richter]

On Thu, 26 Feb 2004 09:31:02 -0500, Peter Hansen <peter at engcorp.com> wrote:

>Leif K-Brooks wrote:
>> 
>> I have a Python file of 1.2MB (it's a static database, not code, so it
>> really does have to be that big). It doesn't seem to get compiled when
>> imported (there's no .pyc file created), so importing it takes a few
>> seconds which I would really like to avoid. Is Python not compiling it
>> because of the large size? Is it because the whole file is on one really
>> long line? How can I fix this?
>
>If it's the main file, it won't get compiled anyway.  Only files
>that are imported are compiled.  In any case, see Aahz' answer for the
>best approach.
>
If your file is highly structured (or you can separately prepare a version that is),
you might consider accessing it via mmap and struct (or even without struct,
if it's e.g., a flat array of fixed-length strings). You could enhance this
by pre-computing an index and storing it at the end or beginning of the file,
maybe data for a dict of (offset,length) values, similarly packed/retrieved
from the raw binary file (byte string) format. If you open the file read-only,
an efficient OS will not even page in or allocate swap file space for anything
you don't access (though 1.2 mb these days is not a big worry ;-).
But if your data is complicated, pretty soon you'd be reinventing pickle
and various db things, and you might as well use what's available already.

Regards,
Bengt Richter