[Tutor] file object in memory

[Monika Jisswel]

I didn't know about tempfile.NamedTemporaryFile ! looks nice because you
don't care about the file anymore, you just use it.
another solution could be to wrap the data (that is each line in the file
still in memory) into mysql statements such as 'insert into table xxxx (aa,
bb, cc) values ( 11, 22, 33)' & pipe it to mysql directly, it will be slow
for tasks like a load but i don't deal with more than 4000 per time & thats
every 15 Minutes.


2008/7/7 Martin Walsh <mwalsh at groktech.org>:

> Monika Jisswel wrote:
> > You know bob, you are very clever, I have used RAM disk for realtime
> > recording of audio before but it never occured to me to use it for light
> > jobs like this one, I just compeletely ignored it as an option & by the
> > way this openes a lot of doors for many of my other programs.
> >
> > but wouldn't it be very nice if there  was a way of creating a file
> > object inside python ? something like :
> > myfile = mkfile('some_string')
> > & we're done ?
>
> If I understand what you're after, perhaps tempfile.NamedTemporaryFile
> is close. Granted it creates a 'real' file, subject to filesystem
> performance, but it is automatically deleted when closed.
>
> ---
>
> Also, while I wouldn't recommend it, I thought it might be worth
> mentioning that you can specify a named pipe (or fifo) with mysql 'load
> data' -- although this may be limited to *nix or Mac, I'm not sure.
>
> It is a tricky dance since, IIUC, the mysql process has to open the
> named pipe for reading before you can write to it, else the write will
> block waiting for a reader and your program will hang. My first thought
> would be to spawn a subprocess of the mysql client, or mysqlimport, to
> start the read, then write data to the named pipe. Something like the
> following (untested):
>
> ...
>
> fifoname = '/tmp/mysql.fifo'
> if os.path.exists(fifoname):
>    os.remove(fifoname)
> os.mkfifo(fifoname)
>
> sql = """\
> "load data local infile '%s' into table statistics
>    fields terminated by '\\t' lines terminated by '\\n';"
> """ % fifoname
>
> conninfo[sql] = sql
> mysqlcmd = \
>  'mysql %(dbname)s -h %(host)s -u %(user)s -e %(sql)s' % conninfo
>
> mysqlp = sp.Popen(mysqlcmd, stdout=sp.PIPE, shell=True)
>
> # should be ready to write
> fifofp = file(fifoname, 'w')
> p1 = sp.Popen(somecmd1, stdout=fifofp, shell=True)
> fifofp.close()
>
> mysqlp.wait()
>
> ...
>
> Seems like an awful lot of work, for questionable gain. I would use a
> regular file. If you're concerned about performance, Bob's ramdisk
> suggestion seems more appropriate.
>
> HTH,
> Marty
>
> >
> >
> >     I don't think there is a pure Pythonic way. Have you considered
> >     using a RamDisk? http://en.wikipedia.org/wiki/RAM_disk for general
> >     details.
> >
> >
> http://www.codeguru.com/cpp/w-p/system/devicedriverdevelopment/article.php/c5789/
> >     for a Windows Ram Disk driver installer. I just tried it - it works
> >     like a charm.
> >
> >     HTH.
> >
> >     --
> >     Bob Gailer
> >     919-636-4239 Chapel Hill, NC
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> http://mail.python.org/mailman/listinfo/tutor
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/tutor/attachments/20080707/f677a0cd/attachment.htm>