(in memory) database

[M.-A. Lemburg]

On 2008-08-31 15:15, mark wrote:
> Hi there,
> 
> I need to extract data from text files (~4 GB) on this data some
> operations are performed like avg, max, min, group etc. The result is
> formated and written in some other text files (some KB).
> 
> I currently think about database tools might be suitable for this. I
> would just write the import from the text files and ... the tool does
> the rest. The only problem I can imagine is that this would not be
> fast enough. But I would give it a shoot.
> Unfortunately I have only some knowledge of SQLite which is not an
> option here.
> 
> Some additional requirements I can think of are:
> - Python (I want to hone my programming skills too)
> - Python-only (no C-lib) for simplicity (installation, portability).
> Therefore SQLite is not an option
> - must be fast
> - I like SQL (select a, b from ...) this would be nice (row[..] + ...
> is a little hard getting used to)
> 
> So far I found PyDBLite, PyTables, Buzhug but they are difficult to
> compare for a beginner.

You could use Gadfly for this since it is pure Python and provides
a standard Python DB-API interface:

    http://gadfly.sourceforge.net/

(the C extensions are optional to speedup processing)

This is the SQL subset it supports:

    http://gadfly.sourceforge.net/sql.html

Another option is SnakeSQL:

    http://pythonweb.org/projects/snakesql/

but I've never used that one, so can't judge its quality.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 01 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611